Identifying and Removing Redundant Xi-Batch Jobs

Why Housekeeping Matters

Xi-Batch systems accumulate jobs over time. Schedules that were once critical fall out of use, one-off jobs remain on the queue in a Done state long after they served their purpose, and jobs that stopped for good years ago sit in the queue looking scheduled. Each job occupies a slot in the shared-memory job table and adds clutter to the queue, making it harder for administrators to see what is actively running.

This article walks through the process of exploring the job queue, working out what the product can and cannot tell you about whether a job is still in use, and safely carrying out the cleanup.

What the Queue Contains

Before deciding anything, it is worth knowing what a healthy queue looks like. A job that runs once, has no repeat specification and is not retained removes itself when it finishes. So the standing contents of the queue are the site's timed and repeating work, plus whatever has been deliberately retained.

That has a direct consequence for housekeeping: the queue is the schedule. Deleting a job deletes the site's only copy of it, because the delete also unlinks the job's script file from the spool directory. There is no undo and no archive unless you take one first.

Exploring the Job Queue

The btjlist command is the primary tool for reviewing jobs from the command line. By default it shows only jobs owned by this host in a terse format. Add -H for column headings and -R to include jobs owned by remote hosts:

btjlist -H
btjlist -HR

The shipped default display shows the job number, user, title, command interpreter, priority, load level, time, condition variables and progress state. Both the default format and the default heading setting can be changed per site or per user through the option-defaults files, so a script should always name the format it wants rather than relying on the default.

Useful Format Codes for Housekeeping

The -F option lets you specify exactly which fields to display. The format string uses % codes, each representing a job attribute, and case matters throughout.

A format string tailored for housekeeping review might look like this:

btjlist -HR -F "%N %U %h %P %r %f %d"

This shows the job number, owner, title, progress state, repeat specification, the time the job last finished, and its auto-delete time.

To filter by queue name, user, or group:

btjlist -H -q "nightly*" -F "%N %h %P %W %r"
btjlist -H -u olduser -F "%N %h %P %f"
btjlist -H -g finance -F "%N %h %P %W"

The -q pattern is a shell-style glob matched against the whole title, and jobs whose title has no colon in it are included as well; -Z excludes those. All three restrictions are applied by the listing itself, which is safer than filtering the output with grep.

What the Product Records, and What It Does Not

This is the section that decides how confident any of the judgements below can be, so it is worth reading before acting on the rest.

Xi-Batch keeps two per-job timestamps that survive a restart: the time the last run started and the time it ended. Both are shown by btjlist. Neither is displayed with a year. The time fields render as hh:mm when the moment is within the next day and as dd/mm otherwise, so a job that last ran a fortnight ago and one that last ran in 2011 produce the same five characters. A blank %f means the job has never run, and that is the only unambiguous reading available from btjlist.

The one place a full date with a year is recorded is the job log, and only if the LOGJOBS system variable has been set. Out of the box it is empty and nothing is logged.

So the honest position is this: unless LOGJOBS was already in use, the product cannot tell you when a job last ran, only what the day and month were. Every conclusion of the form "this has not run for years" is an inference from context - the job's title, the queue it belongs to, who owns it, whether the account still exists - rather than something the system reports. Since the consequence of a wrong inference is destroying a customer's work with no copy anywhere, take a copy first in every case where the answer is not obvious.

Understanding Progress States

The progress state is the single most important field in the listing, and its meaning is stronger than it first appears. The scheduler looks for work only among jobs whose progress state is blank. A job showing anything else is examined only to see whether its auto-delete time has arrived, and is then passed over, however many times its scheduled time comes round.

The states are:

blank (no state)

The job is runnable. It is waiting for its scheduled time, for its conditions to be satisfied, or for capacity. This is the normal state for a healthy repeating job that is not currently executing.

Init, Strt

The job is being started. Init is the first phase, Strt the second.

Run

The job is executing. For a job running on another host the field reads Run:hostname; for a job owned elsewhere and running here it reads Local Exec.

Fin

The job's process has ended and the scheduler is completing it.

Done

The job ran and finished cleanly, and was retained on the queue. It will not run again.

Err

The job's last run exited with a status outside the range configured as normal for it. It will not run again.

Abrt

The job was killed - by an operator, by the maximum-run-time guard, or by the scheduler being stopped while the job was running. It will not run again.

Canc

The job was set to cancelled. It will not run again until someone sets it back.

A repeating job in Err, Abrt or Canc is a fault, not a housekeeping candidate. When a repeating job ends, the scheduler advances its next-run time either way, but resets the progress state to blank only if the job finished cleanly. So a job that failed months ago shows tomorrow's date in the time column and looks perfectly healthy apart from three letters in the progress column. Deleting it as "stale" destroys work that the site still expects to happen. Set it runnable again with btjchange -N, or r in btq, and find out why it stopped.

Identifying Candidates for Removal

With that established, these are the patterns worth looking at. Use a delimiter in the format string and split on it rather than matching whitespace, because the last field of a btjlist line receives no padding and a title can contain anything:

btjlist -HR -F "%N|%U|%h|%P|%r|%f" | awk -F'|' '$4 == "Done"'

Jobs in Done state with no repeat specification

These ran once and were retained. If the %d column is 0 they will stay for ever. This is the largest and safest category.

Jobs in Canc state

A job set to cancelled and forgotten. Check who owns it and whether the cancellation was meant to be temporary before removing it.

Jobs belonging to users who have left

Filter by user, and check the account still exists:

btjlist -HR -u departed_user -F "%N %h %P %r %f"

Jobs whose conditions can never be satisfied

A condition on a variable that has been deleted, or that the job's owner cannot read, holds the job for ever with nothing reported. Compare each job's conditions against the variable list:

btjlist -HR -F "%N %h %C"
btvlist -H

Jobs owned by hosts that are no longer in service

These appear only with -R, and %O gives the originating host. They are the remote host's property; deleting the local copy does not remove the job at its owner.

One pattern that is not a candidate is a job with no scheduled time and no conditions. Such a job runs on the next pass of the scheduler; if it had no repeat specification it has already run and removed itself. A job of that shape sitting in the queue is either in a non-blank progress state, in which case see above, or it is being held back by something else.

Checking When a Job Last Ran

If the LOGJOBS system variable has been set, the scheduler appends one line to the named file for every job event. Set it with btvar, giving the value with -s and the name as the trailing argument:

btvar -S -s /var/log/xi/jobs.log LOGJOBS
btvar LOGJOBS

Each line is pipe-separated: date, time, job number, job title, event, user, group, priority, load level. The date carries a four-digit year, which makes this the only durable record of when a job last ran. The events are lower-case words; the ones that matter for housekeeping are started, completed, error, abort, cancel, auto delete, exceeded runtime and exceeded grace period. Where a request came from another host the event field is prefixed with that host name and a colon.

To find the last time a job completed:

awk -F'|' '$3 == "1420" && $5 == "completed"' /var/log/xi/jobs.log | tail -5

The job number in the log is written without leading zeroes, unlike the zero-padded %N column, so the two do not compare as strings without stripping the padding.

Three cautions. The day and month are written the other way round on machines whose timezone is four or more hours west of UTC. Nothing is recorded when a job is passed over, so silence in the log proves only that the job did not run, never why. And the log begins when LOGJOBS was set - if the answer you need predates that, it does not exist.

Reviewing Variables Before Removing Jobs

Variables and jobs are linked through conditions and assignments. List the variables with btvlist:

btvlist -H

To see which jobs reference a particular variable:

btjlist -HR -F "%N %h %C %S" | grep VARIABLE_NAME

You do not need to work out in advance whether a variable is orphaned, because the scheduler will not let you delete one that is in use. Attempting it gives "Variable %s in use in jobs.", and the same refusal blocks changing the variable's export or cluster flags. So the safe order is to remove the jobs first and then try each variable in turn:

btvar -D VARIABLE_NAME

Two things to know about that check. It counts every job in the queue whatever its progress state, so a long-finished job that will never run again still pins a variable. And the scheduler refuses outright to delete any of the five system variables - LOADLEVEL, CLOAD, MACHINE, LOGJOBS and LOGVARS - so there is no risk of removing one by accident. In btq, V switches to the variable list, C creates a variable and D deletes one.

Backing Up Before Cleanup

Before removing anything, take a copy. Xi-Batch ships four converters that read the scheduler's binary state files and write shell scripts of the commands that would recreate them. On a packaged installation they carry an xb- prefix and live on the ordinary command path.

The scheduler's state files are in the spool directory, by default /var/spool/xi/batch. On a build with XML support - which is every packaged build - the job and variable files carry a suffix, and the converters choose their input format from that suffix, so the name must be given exactly:

cd /var/spool/xi/batch
ls btsched_jfile* btsched_vfile*

To back up jobs, giving the job file, the output script and a directory for the job scripts:

mkdir -p /var/backups/xi/batch/Scripts
xb-cjlist btsched_jfile.xbjl6 /var/backups/xi/batch/Jcmd /var/backups/xi/batch/Scripts

Give the plain name btsched_jfile instead if the installation has no XML support. To produce a single self-contained script with the job scripts embedded as here-documents, use -I with a delimiter word and omit the third argument.

To back up variables, command interpreters and user permissions:

xb-cvlist btsched_vfile.xbvl6 /var/backups/xi/batch/Vcmd
xb-ciconv -D /var/spool/xi/batch cifile /var/backups/xi/batch/Cicmd
xb-btuconv -D /var/spool/xi/batch btufile6 /var/backups/xi/batch/Ucmd

xb-btuconv has no default source directory, so -D is required for it. The others default to the configured spool directory.

When restoring, the order is user permissions, command interpreters, variables, then jobs. This is the reverse of the order above, and it matters: btr rejects a job whose command interpreter or referenced variables do not yet exist.

Two gaps in the backup are worth knowing before you rely on it. xb-cvlist deliberately skips system variables, so LOADLEVEL, LOGJOBS and LOGVARS are not in the script and must be set again by hand. And xb-cjlist does not carry the last-run and last-start times, so restoring from Jcmd loses whatever run history the queue held.

Safely Removing Jobs

Deleting a job unlinks its script file from the spool directory as well as removing the queue entry. Take the copy above first.

Using btq interactively

Move to the job and press D to delete it. If the job is running, kill it first with K, which offers a choice of signal, and wait for it to stop.

Using btjdel from the command line

btjdel takes a list of job numbers, so one invocation handles the whole batch:

btjdel 1420 1421 1422

For a job owned by another host, include the host prefix:

btjdel avon:24918

A running job is skipped with a message unless -y is given, and a job the caller has no permission to delete is skipped with a message as well. Neither of those sets a non-zero exit status, so check the queue afterwards rather than testing the exit code.

Taking a copy of individual jobs first

btjdel -u -k writes a command file and a job file for each job named, into the directory given by -D, and leaves the queue entry in place. This is the per-job equivalent of the whole-queue backup above and is the right step when you are unsure.

Cancelling before deleting

To stop a job running without removing it yet:

btjchange -C 1420

This sets the progress state to Canc, which takes the job out of the scheduler's consideration while keeping it on the queue for review. btjchange -N puts it back.

Unqueueing for archival

In btq, U on the job saves the job script and a command file that could resubmit it. This is the safest approach when you are unsure whether a job might be needed again.

Bulk identification

To list just the job numbers of a user's Done jobs:

btjlist -R -u olduser -F "%N|%P" | awk -F'|' '$2 == "Done" { print $1 }'

That output can be passed to btjdel with xargs, though it is worth reviewing the list first.

Remote Hosts

Xi-Batch reads its host file, by default /etc/xi/batch-hosts, once, when the scheduler starts, and attaches to every host in it that is not marked manual, external or roaming. There is no retry: a host that was unreachable at that moment stays disconnected until either it initiates the connection itself or the scheduler is restarted.

Two consequences for housekeeping. First, nothing reports whether a host is connected. There is no host-listing command, and the only visible evidence is indirect - a disconnected host's jobs and variables disappear from btq and from btjlist -R. Second, the binary does contain connect and disconnect operations, reached by invoking btstart under the names btconn and btdisconn, but the packages do not create those links, so on a packaged installation they are not available and a scheduler restart is the only way to re-attach a host.

Removing a decommissioned host is therefore an edit to the host file followed by a scheduler restart, scheduled for a window in which no jobs are running - stopping the scheduler kills every running job and leaves it in Abrt.

Shared Memory

The job table lives in a shared-memory segment sized by btstart -j, and the variable table by btstart -v. Neither is a hard ceiling: when the table fills, the scheduler allocates a larger segment, copies the contents across and writes a warning to btsched_reps reading "Increased size of job segment to accommodate ...". A submission fails only when the operating system refuses the larger segment.

Deleting jobs returns their slots to the free list at once, so the space is reusable immediately. The segment itself never shrinks, so the memory is not returned until the scheduler is restarted. Note also that btstart's sizing arguments are ignored if the scheduler is already running - resizing means btquit followed by btstart.

Housekeeping Checklist

A periodic review - quarterly or before major upgrades - should cover the following.

Separate faults from redundancy first

List every job in Err, Abrt or Canc that has a repeat specification. These have stopped and were meant to keep running. Fix them; do not delete them.

Review Done jobs with no repeat specification

These are the genuine housekeeping candidates. Check the auto-delete column - a non-zero value means the job will clear itself.

Review job ownership

Identify jobs belonging to accounts that no longer exist.

Inspect conditions and assignments

Ensure that every variable referenced still exists and is readable by the job's owner.

Review the job log

If LOGJOBS is configured, this is the only source that can date a job's last run to the year.

Back up before removing

Run all four converters, and use btjdel -u -k for anything you are unsure about.

Document changes

Keep a record of what was removed and why. The backup scripts are a partial record; the reasoning is not in them.

Best Practices

Schedule housekeeping during quiet periods. Avoid deleting jobs whilst they are executing, since that means killing them first.

Make a check of progress states part of the procedure after every scheduler restart. Every job that was running when the scheduler stopped comes back as Abrt and will not run again until someone notices.

Enable LOGJOBS if it is not already set. It costs almost nothing, it is the only durable record of job activity, and without it a future housekeeping review faces the same evidence problem described above. Remember to set it again after restoring from a backup.

When removing jobs that share a queue name prefix, review the whole queue as a group. Queues usually represent workflows whose members depend on one another through conditions and assignments, and removing one job can leave the rest permanently unable to run.