What the shared memory is for
Xi-Batch keeps its live queue in System V shared memory so that every command can read it without going through the scheduler. The segments are created when the scheduler starts and removed when it stops cleanly.
In normal operation there are two: a job segment and a variable segment, each holding its own header, hash tables and slot array. A third appears while the file monitor is running, and a further one appears briefly while a segment is being enlarged.
The segments are sized at start-up, and they do grow
The scheduler allocates a number of slots - jobs or variables - and the byte size of the segment follows from that number. When the slots run out, the scheduler creates a larger segment at the next key, copies the contents across and removes the old one. Growth is a normal, routine operation and not a last resort.
Growth is nevertheless the thing to size against rather than to rely on. It copies the whole segment while the queue is locked, and a grow that cannot be completed means work is refused.
Setting the initial allocation
btstart takes options, not positional numbers:
| Option | Long form | Meaning |
|---|---|---|
| -j | --initial-job-size | Initial number of job slots |
| -v | --initial-var-size | Initial number of variable slots |
| -l | --initial-load-level | Initial maximum load level - a scheduling limit, not a memory setting |
The shipped service unit runs:
/usr/local/bin/btstart -j 2000 -v 500 -l 10000An option given as zero or omitted falls back to the compiled-in default.
Making the setting permanent
Editing the service unit or the start-up script works, but the setting the product itself regards as the site's own is a system variable in the master configuration file, /etc/xi/batchconfig. That file is read by every program, so the value applies however the scheduler happens to be started.
| Setting | Name in /etc/xi/batchconfig |
|---|---|
| Initial jobs | NUMJOBS |
| Initial variables | NUMVARS |
| Growth step, jobs | INCJOBS |
| Growth step, variables | INCVARS |
Each line is a name, an equals sign and a value:
NUMJOBS = 20000
NUMVARS = 2000The command-line values override the file, so a service unit that passes numbers will win over anything set here. Set the numbers in one place, not both.
The defaults, and why they may not be what you expect
The compiled-in default for variables is 300. The job default is not a single published number: it is derived when the product is built from the shared-memory limit of the build machine, and capped at 20,000. On a current Linux build it reaches the cap.
Because you cannot tell by inspection which value your binary carries, set the job number explicitly rather than relying on the default. Starting the scheduler with no numbers at all does not give a small queue either - the default is likely to be far larger than the 2,000 the shipped service unit passes.
One further rule: a saved queue larger than the number you asked for raises the allocation. At start-up the scheduler sizes the segment from the saved queue file if that is bigger than your figure, so an installation recovering a large queue does not silently truncate it.
How much memory this actually costs
A slot is a fixed-size structure, so the segment is roughly the slot count multiplied by the slot size, rounded up. An Xi-Batch job slot is a large structure, because it carries the job's environment space inside the slot, so a given number of jobs costs considerably more here than the same number would in a print queue.
Read the real figures off your own system rather than estimating them - the bytes column of ipcs -m is the actual allocation:
ipcs -mXi-Batch's segments have keys beginning 0x5869b.
The size column is a Linux one. Solaris, AIX and HP-UX print only the type, identifier, key, mode, owner and group by default; add -b there to get the segment size. ipcs -mb shows the size on all four platforms.
In practical terms, on any machine with a few gigabytes of memory the allocation is small enough that sizing it to cover your peak comfortably costs nothing worth measuring. Sizing it below your peak costs a queue-locking copy at best, and a refused job at worst. Set it generously - but set it from your own peak, not from a table.
What grows, when, and by how much
| Segment | Grows when | Grows by |
|---|---|---|
| Job | A job is created and the slots are full | INCJOBS, by default half the initial-jobs default |
| Variable | A variable is created and the free chain is empty | INCVARS - 20 slots by default |
The default growth step for jobs is large, because it is derived from the large compiled-in default rather than from the number you passed. A site that starts with -j 2000 and fills it does not grow to 2,100; it grows by the full INCJOBS step. Set INCJOBS explicitly if that matters to you.
The variable step of 20 is the exception, and it is small enough that a site adding variables steadily will provoke a whole-segment copy every twenty variables. Size NUMVARS to cover every variable the site will ever hold.
Xi-Batch records each grow in the scheduler report file /var/spool/xi/batch/btsched_reps:
*** Warning: Increased size of job segment to accommodate 12000 jobs (was 2000)
*** Warning: Increased size of variable segment to accommodate 520 vars (was 500)Those lines are the evidence that the initial figure is too small. Search for them before changing anything else.
What happens when the slots run out
Xi-Batch declines the request and keeps running. If the job segment cannot be grown, the job is refused and the program that asked for it prints:
Cannot allocate any more jobsIf the variable segment cannot be grown:
No more space for variables.
I cannot allocate any more shared memory space to hold
new variables - sorry. Please check your kernel configuration.The scheduler stays up and everything already queued continues to run. The cost is the work that was refused, which may be a submission from a script that does not check its exit status.
Growth is a safety net, not a plan
In the shipped release, client programs do not reliably follow every kind of segment growth. Specifically, the address published for readers after a growth is not always the address of the new segment; on Xi-Batch this affects the variable segment. The observable effect is that after a growth of that segment, commands that read the variables may show stale contents or fail to open it, until the scheduler is restarted.
The practical consequence is a simple rule and it is the main advice in this article:
Set the initial allocation to cover your peak, so that the segments never need to grow in normal running. Restarting the scheduler with a larger initial figure is the reliable way to enlarge a queue; growth is what stops a busy day turning into an outage, not what you plan around.
Checking what is allocated now
List the segments:
ipcs -mIdentify them by key: 0x5869b002 and 0x5869b003, plus 0x5869b200 if the file monitor has run. Keys move up by two while a segment is being replaced, so a segment may appear at the next key up after a growth.
The segments are owned by the batch system user, so an owner filter works as well:
ipcs -m | grep batchThe IPC tool reports the same thing more usefully, including which segment is which:
xb-ripcCheck what the scheduler was started with:
ps -ef | grep btschedKernel parameters
Xi-Batch does not read SHMMAX, SHMALL or SHMMNI, and it has no tuning file of its own. On a current Linux kernel the shared-memory limits are effectively unlimited by default and there is nothing to change; the allocations are megabytes, not gigabytes.
Where a limit does bite - an older kernel, or a system where an administrator has lowered the limits deliberately - the behaviour is worth knowing:
- If the kernel rejects the size as too large, the scheduler halves its request and tries again. You end up with fewer slots than you asked for, with no message.
- If the kernel is out of memory or out of segments, the request fails outright and you get the refusal messages above.
- If a segment already exists at the key, the scheduler steps to the next key and retries, up to fifty times. Stale segments from an unclean shutdown therefore do not usually block a start - but they do confuse the client programs, so clear them with xb-ripc -d.
The other Unix platforms Xi-Batch supports - Solaris, AIX and HP-UX - do have shared-memory limits that can be set low, and there the kernel's own documentation is the place to raise them.
Changing the allocation
A change to the initial allocation takes effect at the next start of the scheduler. Stop the product, change the figure, start it again.
btquit -y
btstart -j 20000 -v 10000 -l 10000Make the same change in /etc/xi/batchconfig or in the service unit, or the next reboot will go back to the old figure. Note that stopping the scheduler kills every running job.
Summary
- btstart -j <jobs> -v <variables>, or NUMJOBS and NUMVARS in /etc/xi/batchconfig.
- Set the job number explicitly - the compiled-in default depends on the build machine.
- Size for your peak. The memory cost of a generous figure is negligible; the cost of a segment growing under load is not.
- Xi-Batch refuses new jobs and variables when it cannot grow, and logs each growth to btsched_reps.
- Identify the segments by key in ipcs -m, by the batch owner, or with xb-ripc.