Managing Xi-Batch System Load with Load Levels

What Load Levels Are

A load level is a number attached to a job that stands for the job's impact on the machine. The scheduler adds up the load levels of the jobs it is running and refuses to start a job that would take the total past a limit. It is a finer instrument than a maximum number of jobs, because a large job and a small one count differently.

Three things carry the number.

The job's own load level

An integer stored with the job. A job created under a command interpreter takes the interpreter's load level; only a user with the special create privilege may specify a different one, with btr -l or btjchange -l, and only in the range 1 to 32767.

LOADLEVEL

A system variable holding the total the running jobs on this machine may reach. It accepts a number only.

CLOAD

A read-only system variable holding the total load level the scheduler currently has committed on this machine. The scheduler maintains it; it cannot be set.

Both variables are local to the machine. A network of Xi-Batch hosts has a LOADLEVEL and a CLOAD per host, and a job owned by one host may be run by another whose totals allow it.

The Rule the Scheduler Applies

On each pass over the queue the scheduler walks the jobs in working-priority order and applies two separate tests.

For each job: if the job's own load level plus CLOAD would come to more than LOADLEVEL, this job is skipped and the scan carries on to the next one. Equality is allowed: a job that takes the total to exactly LOADLEVEL starts.

After each local job it starts: if CLOAD has reached or passed LOADLEVEL, the scan stops there and the rest of the queue is not looked at on this pass.

Two consequences follow, and both surprise people.

• A smaller job overtakes a larger one. A job that does not fit is skipped, not queued in front of everything else, so lower-priority jobs with smaller load levels start ahead of it. A large job can be passed over indefinitely on a busy system.
• Nothing is recorded when a job is skipped. There is no log entry, no report and no state change. The job simply stays in the queue with a blank progress column.

LOADLEVEL 20000
CLOAD     15000

Job A, load level 2000  starts   (15000 + 2000 = 17000, not over 20000)
Job B, load level 6000  skipped  (15000 + 6000 = 21000, over 20000)
Job C, load level 5000  starts   (15000 + 5000 = 20000, exactly the limit)

Job B waits for enough of the running work to finish, and the scheduler tries it again on the next pass.

What CLOAD Counts

CLOAD rises by a job's load level when the scheduler starts that job locally, and by a remote-owned job's load level when the scheduler proposes to run it for another host. It falls by the same amount when the job finishes, is killed, is cancelled or dequeued, when a proposal is answered by another host, or when a start fails.

Because it counts jobs from the moment the scheduler commits to them, CLOAD includes jobs in the Init, Strt, Run and Fin progress states, not just those visibly running.

CLOAD is reset to zero every time the scheduler starts. Jobs that were running when it stopped are reloaded as aborted, so they contribute nothing. A restart therefore does not leave load stranded in CLOAD, whatever else it costs.

Reading the Two Values

btvar LOADLEVEL
btvar CLOAD

Each prints the bare number on a line of its own, with no name and no label. To see them side by side with headings:

btvlist -H -F "%N %V" LOADLEVEL CLOAD

Both commands need the scheduler to be running.

These variables are not readable by everyone. They are created owned by the Xi-Batch daemon account and its group, with the default variable modes, which give read and write to the owner, read to the group and neither to anybody else. The group test looks at the caller's primary group only, so a user whose primary group is something else is told

btvar: Unknown variable LOADLEVEL

which is the same message an absent variable produces. Read them as the batch account, or as a user whose primary group is the batch group, or widen the variable's modes deliberately - the article on modes covers how, and the consequences.

Where LOADLEVEL Comes From

LOADLEVEL is held in the saved variable file and survives a scheduler stop, but two things override it.

The compiled-in value used when there is no saved variable file at all - a new installation, or one that has been reinitialised - is 20000.

btstart accepts -l n, which sets LOADLEVEL at startup and overrides whatever was saved. The installed start command uses it. Both the packaged unit file and the start-up script the tarball installer writes run

btstart -j 2000 -v 500 -l 10000

so on a stock installation LOADLEVEL is set to 10000 at every scheduler start, and a change made with btvar lasts only until the next restart. If a different figure is to be permanent, change the number in the start command as well as setting the variable. Check the value actually in force with btstart's own start line - on a systemd installation, systemctl cat xibatch - rather than assuming the compiled-in default.

Changing LOADLEVEL

btvar -s 30000 LOADLEVEL

The -s option takes the value; the variable name is the last argument. Writing it the other way round tries to create a variable called 30000 and fails.

Changing LOADLEVEL requires write permission on the variable, which by default only the owning account has. The change takes effect immediately: raising it lets waiting jobs start on the next pass, and lowering it stops new jobs starting but does not touch jobs already running, so CLOAD may sit above LOADLEVEL until enough of them finish.

CLOAD cannot be set. An attempt is refused with

Attempting to reset a read-only system variable CLOAD

A Job's Own Load Level

A job created without -l takes the load level of the command interpreter it runs under, which is how a site classifies work without giving every user the privilege to set numbers:

btr script.sh                      # the default interpreter's load level
btr -i heavy process-warehouse.sh  # the heavy interpreter's load level

A user holding the special create privilege may override it:

btr -i heavy -l 5000 process-warehouse.sh
btjchange -l 2000 <job_number>

Three things to know about -l:

• The range is 1 to 32767. Outside it the command reports "Load level `n' out of range". (An interpreter's own load level may be set as high as 65535, so a job may inherit a value that -l could not have specified.)
• Without the special create privilege the command reports "Sorry, but you need `special create permission' to respecify load levels." The privilege is checked only when the value differs from what the job already has.
• -i resets the load level from the interpreter, so an -l given before -i is discarded. Put -l after -i.

To see the load levels the queue is carrying:

btjlist -N -F "%L %P %N" | awk '$2 ~ /^(Init|Strt|Run|Fin)$/'

The Per-User Limits

The system-wide test is not the only one. Each user also has a total load level (Totll, 10000 by default), and the scheduler will not start a job if it would take the sum of that user's jobs already under way on this host past it - however much system-wide headroom there is. A second limit, Maxll (1000 by default), caps the load level of any single job the user may create, and is enforced when the job is submitted rather than when it starts. A third, Specll (1000 by default), is only the initial load level given to a new command interpreter.

btulist -H -F "%u %x %t %s" <username>

reads all three; btuchange -T n <user> changes the total. Neither needs a scheduler restart.

The per-user total is the usual reason jobs stop starting while CLOAD sits well below LOADLEVEL. That case has its own article; this one covers the system-wide limit only.

Managing the Total Workload

These are the uses the two variables were designed for. The numbers in them are illustrations of the shape of each technique, not recommendations - what the right figure is on a given machine can only come from watching that machine.

Running less batch work in office hours. Lower LOADLEVEL in the morning and raise it in the evening:

btvar -s 5000 LOADLEVEL
btvar -s 20000 LOADLEVEL

Lowering it also has a second effect worth planning for: with LOADLEVEL at 5000, no job whose own load level exceeds 5000 can start at all, whatever else is running. Setting the daytime figure just under the load level of the heavy work is a way of excluding it by class rather than by count.

Either command can be run from a repeating job, which is how sites automate it. Remember that a job doing this must be able to write the variable, so it has to be owned by an account that may.

Stopping the scheduler while letting jobs finish. Set LOADLEVEL to 0 so nothing new starts, wait for CLOAD to fall to 0, then stop:

btvar -s 0 LOADLEVEL
while [ "`btvar CLOAD`" != "0" ]
do
    echo "Waiting for running jobs to complete..."
    sleep 10
done
btquit -y

btvar prints the bare value, so the command substitution is the whole of it. If the scheduler is restarted afterwards, LOADLEVEL comes back from the start command, so the 0 does not persist.

Starting administrative work when the batch run finishes. A job can carry a condition on CLOAD:

btr -c 'CLOAD=0' nightly-backup.sh
btr -c 'CLOAD<=100' database-maintenance.sh

There must be no spaces around the operator; a space is rejected with "Bad condition". The submitting user must be able to read CLOAD, or the job is refused with "Unknown or unreadable variable CLOAD", so these jobs belong to the batch account or to a user whose primary group is the batch group. Where several administrative jobs are to run, put the CLOAD condition on the first and chain the rest behind it rather than giving each its own; the conditions and assignments article covers chaining.

Letting small administrative jobs through a full system. Choose the load levels so that production work cannot consume the last few units:

Production jobs: load levels in multiples of 100, smallest 100
Administrative jobs: load level 1
LOADLEVEL: 20010

Production totals can only be multiples of 100, so they stop at 20000; the remaining 10 units are reachable only by jobs of 1, which lets up to ten administrative jobs run alongside a full production load. The same idea gives a two-stage shutdown: set LOADLEVEL to 10 to let administrative work continue while production drains, then to 0.

The scheme depends on the load levels being set exactly, which means either a command interpreter per class or the special create privilege, and it is defeated by any production job whose load level is not a multiple of 100. It also does nothing about the per-user total, which applies to the administrative jobs too.

Good Practice

Make the numbers mean something on this machine. The load level has no unit and the product never compares it with anything the operating system reports; it means only what the site decides it means. Whatever scale is chosen, the same kind of work should always carry the same number, or none of the arithmetic above is predictable.

Set them through command interpreters. An interpreter per class of work gives every job of that class the same load level without granting anyone the privilege to choose numbers. The command interpreters article covers how.

Keep the number of distinct values small. Each one is a number somebody has to remember when they add a job; the mixed production and administrative scheme above needs exactly two classes to work.

Write the classification down outside the product. Nothing in Xi-Batch records what a load level of 3000 was meant to represent, and btcilist shows the numbers without the reasoning.

Change LOADLEVEL and the start command together. A change to the variable alone is undone by the next restart.

Check the per-user totals before raising LOADLEVEL. If jobs are not starting, the system-wide limit is often not the one that is binding.