Reinitialising the Xi-Batch Scheduler

Understanding Scheduler Spool Files

Xi-Batch keeps its working state in the spool directory, by default /var/spool/xi/batch. Two files there hold the state the scheduler rebuilds from when it starts:

• btsched_jfile.xbjl6 - the saved job queue
• btsched_vfile.xbvl6 - the saved variables

The .xbjl6 and .xbvl6 suffixes are what the RPM and Debian packages install. A small number of installations built to Xi's order use the plain names btsched_jfile and btsched_vfile. List the spool directory and confirm which pair you have before typing any rm command, because rm -f against the name you do not have succeeds silently and leaves the real file untouched.

The other files in the spool directory are configuration, not state, and a reinitialisation leaves them alone:

• btufile6 - every user's privileges, priority limits and load-level limits
• cifile - the command interpreter definitions
• holfile - the holiday dates
• .xibatch.lic - the licence
• SP files, one per queued job - the job's script, and the only copy of it
• btsched_reps - the scheduler's own report file, in text

Deleting btufile6 destroys every user's privileges. The scheduler rebuilds it with entries for root and the batch account only, both with all privileges, and every other user drops back to the built-in defaults. Deleting cifile rebuilds it with a single sh entry, and every job that named another command interpreter is silently switched to the standard shell as the queue is read in. Deleting holfile creates an empty one, so every holiday date is gone.

The job and variable files are read once at startup. While the scheduler runs they are rewritten no more often than every five minutes, and again when the scheduler is asked to stop. A copy taken from a running system can therefore be up to five minutes behind what the scheduler is holding in memory.

When Reinitialisation Is Appropriate

Reinitialisation returns the scheduler to a fresh state by discarding the saved job queue and the saved variables and letting the scheduler rebuild them. It is appropriate when:

• Testing scheduler configuration changes on a system whose queue is disposable
• Starting a new operational phase
• Recovering from a corrupt job or variable file

A corrupt file announces itself: the scheduler refuses to start and writes Invalid format of XML saved job file or Invalid format of XML saved variables file to btsched_reps. That failure happens after the scheduler has created its request socket, and it exits without removing it, so clear the leftover socket with xb-ripc -d before each retry or the next start fails for a different reason. Try to salvage the file with xb-cjlist before discarding it, as described under Backup First.

Moving a site to new hardware is a backup and restore job rather than a reinitialisation: the point of the move is to carry the schedule across, and the tools below are the way to do it.

This is destructive in two ways that are easy to underestimate. First, a job that runs once and carries no repeat time removes itself from the queue when it finishes, so what accumulates in the queue is the site's timed and recurring work - the schedule itself. Discarding the job file discards that schedule, and the SP script files left behind in the spool directory are no longer reachable through any command. Second, stopping the scheduler kills every job that is running at the time with SIGKILL, immediately and with no grace period. Choose a quiet moment and take the backup first.

Backup First

Stop the scheduler before you copy anything, so that what is on disk is what the scheduler was holding:

# Stop the scheduler (this kills any running jobs)
systemctl stop xibatch          # packaged Linux installations
# btquit -y                     # everywhere else

# Raw copy of the whole spool directory
backup=/var/spool/xi/batch.backup.$(date +%Y%m%d-%H%M%S)
mkdir -p "$backup"
cp -pR /var/spool/xi/batch/. "$backup"/

A raw copy restores only onto a compatible installation. For a backup you can read, edit and replay - and the only kind that survives a move to different hardware - use the product's own conversion tools. Each writes a shell script that recreates what it read. Run them as root or as the batch account:

mkdir -p /var/backups/xibatch/Scripts
cd /var/backups/xibatch

xb-cjlist  -D /var/spool/xi/batch btsched_jfile.xbjl6 joblist.sh Scripts
xb-cvlist  -D /var/spool/xi/batch btsched_vfile.xbvl6 varlist.sh
xb-ciconv  -D /var/spool/xi/batch cifile cilist.sh
xb-btuconv -D /var/spool/xi/batch btufile6 userlist.sh

Give xb-cjlist and xb-cvlist the file name your spool directory holds, suffix included. They choose their input format from the suffix, so passing btsched_jfile on a libxml2 build fails with Sorry cannot open btsched_jfile. The third argument to xb-cjlist is a directory that must already exist; the job scripts are copied into it and joblist.sh refers to them there. Restore in the order users, command interpreters, variables, jobs, because jobs are rejected if the variables and command interpreters they name are missing.

Tarball installations also carry xb-backup, an interactive wrapper that asks what to save and runs these four tools for you. It is not part of the RPM or Debian packages.

Reinitialisation Procedure

# 1. Stop the scheduler and confirm it has gone
systemctl stop xibatch          # packaged Linux installations
# btquit -y                     # everywhere else
ps -ef | grep btsched

# 2. Remove the saved job and variable files, using the names
#    your spool directory actually holds
rm -f /var/spool/xi/batch/btsched_jfile.xbjl6
rm -f /var/spool/xi/batch/btsched_vfile.xbvl6

# 3. Start the scheduler again
systemctl start xibatch         # packaged Linux installations
# btstart                       # everywhere else

# 4. Check the queue is empty and the scheduler is answering
btjlist -H

btquit -y is what the packaged systemd unit runs for systemctl stop, so the two are the same operation. It does not return until the scheduler has finished: the scheduler halts the queue, shuts the network side down, waits about ten seconds in total, writes the job and variable files out and only then exits, closing the connection btquit is waiting on. The command needs the stop scheduler privilege, which is not one of the default user privileges and which by default only root and the batch account hold; without it the scheduler answers Sorry - you (name) are not allowed to stop the Xi-Batch scheduler - after it has already halted the queue and killed the running jobs.

Never use kill -9, pkill -9 or SIGKILL on btsched. Everything that makes this procedure safe happens in the shutdown path, and SIGKILL skips all of it: the job and variable files are left as they were, the shared memory segments and the request socket are left behind, and on a packaged Linux installation the unit's Restart=on-failure brings the scheduler back five seconds later, in the middle of your rm commands. Sending SIGKILL a couple of seconds after asking for a clean stop is the worst case of all, because that is exactly when the scheduler is partway through the shutdown and has yet to write its files. If the scheduler will not respond to btquit, send it a plain kill (SIGTERM), which it catches and treats as a request to shut down cleanly.

After a clean stop there are no IPC resources to clear: the scheduler removes its own shared memory segments, request socket and lock files as it exits. Clear them by hand, with xb-ripc -d, only when the scheduler was killed or crashed - a leftover request socket makes the next start fail. The article Managing Xi-Batch IPC Resources covers that case.

Removing the job file orphans every SP script file in the spool directory: nothing refers to them once the queue is gone. They are harmless, and they are also the last trace of what the schedule was, so leave them until you are satisfied the new state is right.

What Happens During Restart

Starting without the saved files, the scheduler:

1. Creates an empty variable file and initialises the system variables: the maximum load level (from -l, otherwise 20000), the current load level, the job log and the variable log - the last two as empty strings - plus the machine name when the licence permits networking
2. Creates cifile with a single sh entry, and an empty holfile, if those files are missing
3. Creates an empty job file
4. Creates its IPC resources: two shared memory segments, the request socket and two lock files
5. Records btsched started. in btsched_reps and begins accepting job submissions

The segments are sized from btstart -j (jobs) and -v (variables), and the initial maximum load level from -l. A bare btstart falls back to the NUMJOBS and NUMVARS settings in /etc/xi/batchconfig. The packaged systemd unit starts the scheduler as btstart -j 2000 -v 500 -l 10000, so on those systems start it with systemctl start xibatch rather than by hand, or the site's sizes and load level will not be the ones it normally runs with.

After Reinitialisation

The system is in a clean state:

• The job queue is empty, and the SP script files of the jobs that were in it are orphaned
• Every user-defined variable is gone, and the system variables are back at their defaults
• Job and variable logging is switched off, because the log variables are recreated empty
• User privileges, command interpreters and holidays are as they were, provided you left those files in place
• The scheduler is ready for job submissions

Resubmit the jobs you want from the backup: joblist.sh recreates them along with the scripts saved beside it, and varlist.sh should be run first so that the conditions and assignments in the jobs resolve. If you took only the raw copy, the same conversion tools will read the copied files - point their -D at the backup directory.

Preserving the Activity Log

btsched_reps is the scheduler's own report file. It records starts and stops, licence messages and panics, each entry timestamped, and the scheduler appends to it and never reads it, so leaving it in place across a reinitialisation costs nothing. Nothing in the product rotates or trims it; it grows for as long as the installation runs. To reclaim the space, truncate it rather than delete it - the scheduler holds the file open, so a deleted file goes on being written to where you cannot see it until the scheduler is restarted.

The record of what jobs actually did is a different file. It is written only when the job log system variable holds a file name, and that variable lives in the variable file, so a reinitialisation empties it and job logging stops. If the site logs job activity, note the job log and variable log settings before you start and set them again afterwards.