Managing Xi-Batch IPC Resources

How Xi-Batch Uses IPC

Xi-Batch coordinates the scheduler btsched and the client commands through a small set of interprocess-communication objects. Exactly which objects exist is fixed when the product is built, and every build Xi Software packages uses the same configuration:

• Two System V shared memory segments - one holding the live job queue, one holding the live variables. This is the queue state the commands read.
• Two lock files - btjob.lock and btvar.lock in the spool directory, used with flock to serialise access to those segments. The packaged build uses file locking in place of a semaphore set.
• One Unix domain socket - btsched_req, in the sockets directory (default /var/spool/xi/btuds, overridable with SPUDSOCKS in /etc/xi/batchconfig). Client commands send their requests to the scheduler over this socket. The packaged build uses it in place of a System V message queue.
• One semaphore set of three - created only when the licence enables networking, and used solely to coordinate variable locking between hosts.

On a standard installation, therefore, ipcs shows two shared memory segments belonging to Xi-Batch, no message queues, and one semaphore set on a networked licence or none on a standalone one.

A small number of installations run a variant built to Xi's order, and those use different mechanisms: memory-mapped files btmm_jobs and btmm_vars in the spool directory in place of shared memory, so ipcs shows no Xi-Batch shared memory at all; a semaphore set in place of the two lock files; or a System V message queue in place of the request socket, with an extra shared memory segment alongside it. You do not need to know which you have in advance - run xb-ripc and it reports what it finds on your system.

How the keys are formed. The keys are compiled-in constants rather than values derived with ftok, so they are the same on every host. All of them begin 0x5869b:

• Job shared memory segment - 0x5869b002
• Variable shared memory segment - 0x5869b003
• Network-locking semaphore set - 0x5869b003 (the same number in the separate semaphore key space)
• Message queue, on an installation that uses one - 0x5869b000, with its transport segment at 0x5869b100
• Lock semaphore set, on an installation that uses one in place of the lock files - 0x5869b001

The two segment keys are a starting point rather than a fixed value. If a key is already taken, or when a segment is replaced by a larger one, the scheduler steps the key on by two and tries again, wrapping after a hundred keys. A segment can therefore appear anywhere from 0x5869b002 to 0x5869b065.

Multiple environments. Where several independent Xi-Batch environments run on one host, the environment number in XIBATCH_ENV shifts every key: the value added is (environment number + 1) multiplied by 1024. Environment 0 puts the job segment at 0x5869b402, environment 1 at 0x5869b802, environment 2 at 0x5869bc02, and environment 3 at 0x5869c002 - past the 0x5869b prefix altogether. On such a host, select entries by their owning user or let xb-ripc work the keys out for you, rather than matching on the key text.

Normal Operation

The scheduler creates these objects when it starts and removes them when it stops cleanly, whether it is stopped with btquit or sent SIGTERM. The lock files are left in place between runs and truncated at the next start.

The initial job and variable allocations come from NUMJOBS and NUMVARS in /etc/xi/batchconfig, or from the -j and -v arguments to btstart. When the queue outgrows its allocation the scheduler creates a larger segment at the next key, copies the contents across and removes the old one, logging Increased size of job segment to btsched_reps. Seeing three segments for a moment during that changeover is normal.

Checking IPC Resources

# What Xi-Batch itself finds. Run as root or as the batch user.
xb-ripc

# The shared memory segments, selected by owner
ipcs -m | grep batch

# The request socket. Its presence is how every Xi-Batch command
# decides whether the scheduler is running.
ls -l /var/spool/xi/btuds/btsched_req

# The lock files
ls -l /var/spool/xi/batch/btjob.lock /var/spool/xi/batch/btvar.lock

Selecting by owner rather than by key is the more reliable filter: it survives the key stepping described above, and it keeps Xi-Text out of the listing (Xi-Text keys begin 0x58691). The administration manual writes ipcs -o. On Solaris, AIX and HP-UX that option adds the outstanding-usage figures - for a shared memory segment, the number of processes attached to it - rather than the owner, which those platforms print by default. Linux has no -o at all, and a plain ipcs -m gives you both the owner and the attach count in its nattch column.

One pair of entries belongs to something else. A shared memory segment at 0x5869b200 and a two-member semaphore set at 0x5869b201 belong to btfilemon, the file monitor, which is a separate program run by an ordinary user rather than by the scheduler. Those two keys carry no environment offset and are owned by whoever started the monitor, so an owner filter will miss them and a key filter will catch them. Leave them alone: xb-ripc ignores them, and the monitor removes them itself when the last one exits. The file monitor is installed only by the tarball installation, so a package install will never show them.

When Resources Need Cleanup

After a power failure, or after btsched has been killed with SIGKILL rather than stopped, the shared memory segments, the network semaphore set and the btsched_req socket file are all left behind.

The socket file is the one that prevents a restart. The scheduler binds that path at startup, and if the file already exists the bind fails and btsched exits immediately with status 0 and no message. btstart reports that it is restarting the scheduler, and nothing comes up.

Stale shared memory does not by itself stop the scheduler starting - it simply allocates at the next free key. It confuses the client commands instead, because they scan the key range for a segment and can find the abandoned one ahead of the live one.

Symptoms indicating stale resources:

• btstart reports that it is restarting the scheduler, but no btsched process appears
• Commands report that the scheduler is not running when it should be
• Commands show a stale or empty job queue, or report Panic! Cannot read job queue
• More than two Xi-Batch shared memory segments are present while no scheduler is running

Cleanup Procedure

Stop the scheduler first: xb-ripc -d removes whatever it finds, including the live queue, so it must never be run against a running system.

# On a packaged Linux install, stop the service rather than the
# program: the unit runs btquit -y for you, and it is configured to
# restart btsched on failure if you kill it behind systemd's back.
systemctl stop xibatch

# On other platforms, or where the service is not managed by systemd
btquit -y

# Check it has gone
ps -ef | grep btsched

# If a btsched process remains, send it SIGTERM - never SIGKILL.
# SIGTERM runs the same clean shutdown as btquit and removes the IPC
# objects; SIGKILL is what leaves them behind.
kill PID

# Remove anything left over. Run as root or as the batch user.
# This also deletes the stale btsched_req socket.
xb-ripc -d -o /dev/null

# On a host running multiple environments, name the environment
xb-ripc -d -S 1 -o /dev/null

# Confirm nothing of ours remains
ipcs -m | grep batch
ls -l /var/spool/xi/btuds/btsched_req

# Restart
systemctl start xibatch

# Verify: the request socket is back and the queue reads
ls -l /var/spool/xi/btuds/btsched_req
btjlist -H

Where xb-ripc is unavailable, ipcrm will remove the segments by identifier. Take the identifier from the second column of the ipcs listing on Linux and from the ID column on Solaris, AIX and HP-UX, remove only entries owned by batch, leave the two file-monitor entries described above in place, and delete the btsched_req socket file by hand afterwards. An entry cannot be removed while a process is still attached to it, so make sure the user commands have exited as well as the scheduler.

After cleanup, the scheduler creates fresh IPC resources and reloads the queue from the saved job and variable files, and normal operations resume.