Controlling the Xi-Batch Job Start Rate with STARTLIM and STARTWAIT

Understanding Job Start Rate Control

When many jobs become ready at the same moment - scheduled for the same time, or released together when one variable changes - the scheduler can find itself starting a large number of jobs simultaneously. STARTLIM and STARTWAIT bound how many it has in the middle of starting at any one time.

Both are settings in the Xi-Batch master configuration file, /etc/xi/batchconfig. They are read once, when a program starts, so a change reaches the scheduler only when the scheduler is restarted.

They are not system variables. Older documentation presents STARTLIM and STARTWAIT as variables that btvar can read and set, and says that external packages can query and modify them that way. The scheduler creates no variables of those names, and it never consults a variable when deciding how many jobs to start. A command of the form btvar -s 10 STARTLIM fails with a variable-not-found error, because there is no such variable; adding -C creates an ordinary user variable of that name, which has no effect whatever on job starting. Use the configuration file.

The Problem

Starting a job is not free. For each one the scheduler locks the variables the job depends on, hands the job to one of its execution processes, and that process forks and execs the job. The job itself then does whatever it does at startup - opening files, connecting to a database, contacting another host.

Resource pressure at start time occurs when:

• Many jobs are scheduled for the same time, such as midnight
• One variable change releases a large number of waiting jobs together
• Network-intensive jobs all establish their connections at once
• Startup I/O from many jobs arrives at the storage subsystem together

Symptoms:

• Load average spikes at job start times
• Network timeouts during a burst of job starts
• Jobs failing during their own initialisation rather than in their main work

What these settings do not control. STARTLIM limits jobs that are being started, not jobs that are running. The number of jobs running concurrently is governed by the LOADLEVEL system variable and the load value carried by each job. If the machine is saturated by work that is already under way, LOADLEVEL is the control to reach for, and lowering STARTLIM will not help.

How STARTLIM Works

STARTLIM

The maximum number of jobs the scheduler will have in the process of starting at any one moment

Default value: 15

The scheduler keeps a count of jobs it has begun to start and not yet had confirmed as started. A job joins that count when the scheduler hands it to an execution process, or - for a job that may run on another host - when the scheduler proposes it to that host. It leaves the count when the job is confirmed started, when the proposal is answered, or when the start fails.

Each time the scheduler reviews the queue it walks the jobs in queue order, which is priority order, and for each job that is otherwise ready to run:

1. If the count of jobs already being started has reached STARTLIM, the scheduler stops reviewing the queue there and arranges to come back to it.
2. Otherwise it starts the job, adds it to the count, and moves on to the next job.

A job normally leaves the count within a fraction of a second, so on an unloaded system the limit is reached only in bursts, and the scheduler resumes as soon as the first of those jobs reports that it has started.

How STARTWAIT Works

STARTWAIT

The longest the scheduler will wait, in seconds, before reviewing the queue again after STARTLIM stopped it

Default value: 30 seconds

STARTWAIT is a ceiling on the wait, not a fixed pause between batches. The scheduler reviews the queue again whenever anything happens that might change it - and a job reporting that it has started is exactly such an event. In ordinary operation the scheduler therefore carries on well within a second of jobs clearing the count, and STARTWAIT never elapses.

What STARTWAIT provides is the backstop: if the jobs being started are slow to report back, or something has gone wrong and they never do, the scheduler still returns to the queue after this many seconds rather than waiting indefinitely. Other pending work - a job due to run, a time limit about to expire - can bring it back sooner.

Checking Current Settings

No Xi-Batch command reports these values. They are not variables, so btvar and btvlist do not show them, and no btq screen displays them. The only place to read them is the configuration file itself:

grep STARTLIM /etc/xi/batchconfig
grep STARTWAIT /etc/xi/batchconfig

The installation process writes both settings into the file with their descriptions. A setting whose value is the built-in default is written commented out, so a freshly installed file looks like this:

# Start limit.
# This is a limit to the number of jobs which can be started at once.
# STARTLIM = 15

# Start wait.
# This gives a wait time between each block of jobs to be started.
# STARTWAIT = 30

Read it as follows:

• A line STARTLIM = 20 - the value in force is 20.
• A line # STARTLIM = 15, or no line at all - the built-in default is in force, which is 15 for STARTLIM and 30 for STARTWAIT.

Note that the file records what the next scheduler start will use. If the file has been edited since the scheduler was started, the running scheduler is still using the previous value and there is no way to interrogate it.

Adjusting STARTLIM

Consider raising STARTLIM when:

• Jobs are lightweight and complete their startup quickly
• Bursts of ready jobs take noticeably long to all reach the running state
• No resource contention is observed at start times

Consider lowering STARTLIM when:

• Load spikes sharply at the moment a group of jobs becomes ready
• Network or storage saturates during a burst of starts
• Jobs fail during their own initialisation when started in a group

Adjusting STARTWAIT

STARTWAIT has an effect only when jobs are slow to confirm that they have started, since otherwise the scheduler resumes before it elapses. Lowering it makes the scheduler retry sooner in that case; raising it makes it retry later. Neither changes how many jobs start at once - that is STARTLIM alone.

Leave STARTWAIT at its default unless you have a specific reason: a machine where job startup is genuinely slow and the scheduler is seen to be idle waiting, or one where the retry itself is unwelcome.

Applying a Change

Edit /etc/xi/batchconfig as root and uncomment or add the setting. Use an equals sign: a colon is accepted by the running product, but the installation process only recognises the equals form, so a setting written with a colon is lost the next time a package upgrade rewrites the file.

STARTLIM = 8
STARTWAIT = 30

The value must be a positive whole number. Zero, a negative number or anything that is not a number causes every Xi-Batch program - not only the scheduler - to print a message naming the file and the line and exit immediately. Check the file before restarting.

The scheduler reads the file only at startup, so the change takes effect when it is restarted. On a packaged Linux installation:

systemctl restart xibatch

Otherwise stop and start it by hand:

btquit -y
btstart

Stopping the scheduler kills the jobs that are running. btquit sends SIGKILL to every locally running job with no grace period; the queue entries survive, the work in progress does not. Restart the scheduler during a quiet period, and confirm with btjlist that nothing is running first.

Finding a Suitable Value

There is no value that is right for a given workload in the abstract - what matters is how long each of your jobs takes to get from being handed to an execution process to reporting that it has started, and that depends on the job. Work from observation:

Step 1: Watch a real burst. Identify a time when a group of jobs becomes ready together, and observe the system across it.

uptime
btjlist

Step 2: Change one setting at a time. Halve STARTLIM if starts are swamping the machine; the effect is on the size of the burst, and it should be visible on the next occurrence.

Step 3: Watch the same burst again, across several days before concluding anything. Job mixes vary by day of the week and by month end.

Step 4: Record what you changed and why, outside /etc/xi. Nothing in the product records it, and a package upgrade rewrites the configuration file from its own template - your values are carried over, but any comment you added to the file is discarded.

Two related settings in the same file bear on the same behaviour and are worth knowing about before adjusting STARTLIM:

• EXECPROCS - the number of processes the scheduler runs to start jobs, three by default. This is what limits how quickly the jobs allowed by STARTLIM are actually got under way.
• LOADLEVEL - a system variable rather than a configuration setting, and the control over how many jobs run concurrently. See the article on load levels.

Example: A Large Group of Network Jobs

400 jobs are scheduled for midnight, each of which opens a connection to another host. At the defaults, the scheduler starts jobs from the front of the queue until 15 of them are in the process of starting, then stops. As each of those confirms that it has started, the scheduler is woken and starts another. The 400 jobs enter the running state in a rolling stream of at most 15 concurrent startups, not in one burst of 400.

How long that stream takes depends entirely on how long these jobs take to start - with the connection setup on the far side of it, that may be seconds each rather than milliseconds. If the far host or the network is what is being overwhelmed, lowering STARTLIM narrows the stream. If instead the problem is 400 jobs all running at once once they have started, STARTLIM will not help and LOADLEVEL is the setting to use.

Troubleshooting

Jobs are ready but not starting

Check LOADLEVEL and CLOAD first - the load level is the more common cause. STARTLIM bounds only the burst, not the total number of running jobs.

btvar LOADLEVEL
btvar CLOAD

Jobs still all start at once

Confirm the setting is uncommented in /etc/xi/batchconfig, and that the scheduler has been restarted since the edit. Until it is restarted the running scheduler is using the previous value, and nothing reports that.

An Xi-Batch command exits immediately with a configuration error

The value given to STARTLIM or STARTWAIT is not a positive whole number. Every Xi-Batch program reads this file and refuses to run, so this affects the whole system and not only the scheduler. Correct the line named in the message.

btvar shows a STARTLIM variable

Someone has created an ordinary variable of that name, most likely by following older advice to set it with btvar. It has no effect on job starting and can be deleted with btvar -D STARTLIM.

Best Practices

Change one setting at a time

Each change costs a scheduler restart, and a restart kills running jobs. Batch the edits, restart once, and observe.

Observe before adjusting

Collect data on system behaviour across a real burst before changing anything.

Keep your own note of changes

Record the value and the reason somewhere outside /etc/xi. Nothing in the product keeps that history, and an upgrade rewrites the file.

Restart during a quiet period

Check with btjlist that no jobs are running before stopping the scheduler.

Keep a copy of the file

An upgrade carries your values across but rewrites the file; a full package removal deletes it, and a reinstallation writes a fresh one with the defaults.

Reach for LOADLEVEL for sustained load

STARTLIM addresses the burst at start time only.