Why an Xi-Batch Job Will Not Start When CLOAD Is Below LOADLEVEL

Observations include - jobs start late, start in an unpredictable order, or fail to run at all, whilst every system-wide indicator appears healthy.

Symptoms typically reported:

• Mission-critical jobs start minutes or hours after their scheduled time
• The same job starts at a different time each day for no apparent reason
• Daily jobs occasionally do not run at all
• Forcing a job with f or g reports that the job is running when it plainly is not
• CLOAD is well below LOADLEVEL, so there is apparently spare capacity
• Raising LOADLEVEL makes no difference

The system-wide load level is one of nine tests the scheduler applies to each job on every pass through the queue, and it is the only one of the nine that any command reports. A job held back by any of the other eight looks exactly like a job waiting its turn: nothing is written to the report file, nothing is written to the job log, and no message is sent to anyone.

What the scheduler tests, in order

The scheduler walks the whole queue in working-priority order and applies these tests to each job in turn. The first one that fails ends the scheduler's interest in that job until the next pass.

Two properties of this list explain most of the confusion.

Test 2 skips the job and carries on

A job that would take CLOAD over the limit is passed over, but the scan continues, so a smaller job further down the queue starts ahead of it. The bigger job can be overtaken repeatedly. Tests 8 and 9 behave differently - they abandon the rest of the scan altogether, so jobs behind them are not even looked at until the next pass.

Nothing is recorded when a job is passed over

There is no log entry, no report-file line and no message for any of the nine tests. The job log records events that happened; a job that did not start produced no event. A job can therefore be skipped every minute for a year and leave no trace.

The second load limit: the owner's total load level

Test 6 is the one that matches the symptom in this article's title, because it is a load limit that is entirely independent of CLOAD and LOADLEVEL and is reported by nothing.

Each time the scheduler considers a job it re-scans the whole queue and adds up the load levels of every job with the same owning user id that is currently in progress state Init, Strt, Run or Fin and running on this machine. If that total plus the candidate job's own load level exceeds the owner's total load level, the job is skipped.

Three per-user limits exist. The shipped defaults are:

With the shipped defaults a job costs 1000 and a user's total is 10000, so every user is capped at ten concurrent jobs whatever LOADLEVEL says. On a stock installation LOADLEVEL is also 10000, so the two limits bind together for a single busy user and the per-user limit bites first as soon as a second user is active.

The total load level is held as an unsigned 16-bit value, so it cannot be set above 65535. On a system whose LOADLEVEL has been raised into the hundreds of thousands, no user can ever be given a matching allowance, and the per-user limit becomes the binding constraint permanently.

Read the three limits with btulist:

btulist -H -F "%u %x %t %s"

The first line of the listing is the DEFAULT entry, which applies to every user without an entry of their own. Change a user's total with btuchange:

btuchange -T 20000 fred

The interactive program btuser shows the same table, with t and m to set a selected user's total and maximum, and T and M to set the defaults applied to new users. Changes are saved on exit.

A job whose progress code is not blank never starts

Test 1 is the cause of "the job simply did not run", and it is the one most often mistaken for a load problem.

The scheduler looks for work only among jobs whose progress code is blank. A job showing Done, Err, Abrt or Canc is examined only to see whether its auto-delete time has arrived, and is then passed over. That is true however many times its scheduled time comes round.

What makes this hard to spot is that the schedule keeps advancing. When a repeating job ends, the scheduler advances its next-run time either way, but it resets the progress code to blank only if the job finished cleanly. A job that exits outside its normal range is left in Err with tomorrow's date in the time column, so btjlist shows a job that is apparently scheduled and healthy, and it will never run again.

Forcing such a job does not work either. The force request is refused for any job whose progress code is not blank, and the refusal message is "Job is running", which is misleading for a job stuck in Err or Canc.

To find them:

btjlist -HR -F "%N %U %h %P %W %r %x"

Any job with a repeat specification in the %r column and Done, Err, Abrt or Canc in the %P column is parked. The %x column gives the exit code from the run that parked it. To make one runnable again:

btjchange -N 1420

In btq, r on the job does the same thing, and P offers the full list of progress codes.

How a job comes to be parked

Every one of these leaves the job with a non-blank progress code and no further attempt to run it:

• The job exited with a status outside the range configured as normal for it, giving Err
• The job was killed, by an operator or by the maximum-run-time guard, giving Abrt
• The scheduler could not fork a process for the job, giving Err
• Someone set it to cancelled and did not set it back, giving Canc
• The job ran once, had no repeat specification, and was retained, giving Done
• The scheduler was stopped, or the machine was rebooted, while the job was running

The last of these deserves stating plainly. Stopping the scheduler sends SIGKILL to every job running on this machine. When the scheduler is started again it reads the saved job file, and every job that was recorded as starting or running is loaded back as Abrt. So each scheduler restart parks every job that was running at the time, permanently, and the queue afterwards looks entirely normal apart from the progress column.

Checking for parked jobs is therefore part of the restart procedure, not an occasional audit.

What a restart does not do

A scheduler restart does not leave load stranded in CLOAD. CLOAD is a counter held in the shared-memory segment and it is set to zero every time the scheduler starts, after the saved variable file has been read. Jobs that were running come back as Abrt, which also removes them from the per-user total, since that sum counts only progress states Init, Strt, Run and Fin.

So a CLOAD figure that never falls to zero at a quiet period is telling you about work the scheduler currently believes is in progress, within this run of the scheduler. The candidates are a job whose process genuinely never exits - a service submitted as a batch job - or a job proposed to a remote host that has not yet answered, since an outstanding proposal is counted in CLOAD too.

LOADLEVEL is also reset at every start. The installed start command supplies it explicitly, so the value in force after a restart is the one on the start command line rather than anything set with btvar since. This is covered in the load-levels article.

Diagnosis

Work down the nine tests. The order below is the order that finds the answer fastest.

1. Rule out the parked-job case first. It is the most common and the cheapest to check:

btjlist -HR -F "%N %U %h %P %W %r %x"

A job with a repeat specification and a non-blank progress code has stopped for good. That is the whole diagnosis; nothing about load is involved.

2. Read the two system variables. Each btvar command takes one variable name and prints the bare value:

btvar CLOAD
btvar LOADLEVEL

If either reports "Unknown variable", the account you are using cannot read it rather than it being absent - the two variables are created with the batch daemon's default variable modes, which give read access to the owner and the primary group only.

3. Do the per-user arithmetic. List what the scheduler counts against the owner, which is that user's jobs in Init, Strt, Run or Fin running locally:

btjlist -HR -u fred -F "%N %h %P %L %i %b"

Add the %L column for the rows whose %P column is Init, Strt, Run or Fin, and compare the total with that user's Totll from btulist. If the remaining headroom matches the concurrency you observe in practice, test 6 is the limit being hit.

4. Check the conditions. A job with conditions waits silently until every one is satisfied:

btjlist -HR -F "%N %h %C"
btvlist -H

Compare each condition's variable name and comparison against the variable's current value. A condition on a variable that no longer exists, or that the job's owner cannot read, holds the job for ever.

5. Check the scheduled time and the missed-occurrence rule. A repeating job whose due time passes while it is held back is handled according to its reschedule option. Where that option is set to skip, and the delay exceeds the margin the scheduler allows, the occurrence is abandoned: the time advances to the next occurrence, the job moves to the back of the queue, and nothing is recorded. That is the mechanism behind "daily jobs occasionally do not run at all" on a system that is otherwise healthy.

6. Consider the start rate last. STARTLIM caps the number of jobs in the process of starting at one instant, and when the cap is reached the scheduler abandons the rest of the scan. It delays starts; it does not prevent them. It is a master-configuration setting rather than a variable, and no command reports the value the running scheduler is using - see the article on STARTLIM and STARTWAIT.

Confirming from the job audit trail

Where LOGJOBS is set the log settles several of these questions, because it is the only place in the product that records a full date including the year.

Each line is pipe-separated: date, time, job number, job title, event, user, group, priority, load level. The events relevant here are started, completed, error, abort, cancel, exceeded runtime and exceeded grace period.

To read one job's history, reading the file whose name is the value of LOGJOBS:

awk -F'|' '$3 == "1420"' `btvar LOGJOBS`

The job number in the log is written without leading zeroes, which is a different form from the zero-padded %N column btjlist prints. A remote job appears as host:number. Matching the two by string comparison therefore needs the padding stripped.

Two things the log cannot tell you. It records only what happened, so a job that was passed over produces no line at all and its absence proves nothing on its own. And the log is written by the scheduler from the moment LOGJOBS is set, so it says nothing about the period before that.

Where it does settle the question is concurrency: count how many of one user's jobs are between a started and a completed line at any moment. If that count never exceeds a fixed number over days of operation, that number is the user's headroom rather than a coincidence.

Clearing and preventing

Reset a parked job

btjchange -N on the job number, or r in btq. Confirm the job's process is genuinely absent first if the progress code is Run - resetting a job whose process is still alive can result in two copies running.

Release capacity in a controlled window

Making a large number of jobs runnable at once will let them all start together. Either work in small batches, or lower LOADLEVEL temporarily so that the backlog drains gradually.

Set a maximum elapsed run time

In btq, u opens the process parameters for a job and R sets the maximum permitted elapsed run time. An over-running job is then killed automatically with the signal set by K, after the grace period set by g. This is the only mechanism that ends a job which never exits.

Review the queue after every scheduler restart

Any job running when the scheduler stops comes back as Abrt and never runs again. Make a check of progress codes part of the restart procedure.

Set the reschedule option deliberately

A repeating job set to skip missed occurrences will silently drop one whenever it is held back past the margin. Where every occurrence matters, choose catch-up instead.

Enable the job audit trail

Setting LOGJOBS costs almost nothing and is the only durable record of when a job last ran. Note that the backup tool xb-cvlist deliberately omits system variables, so LOGJOBS is re-set by hand after a restore.

Keep services out of the queue

A process intended to run continuously holds its load level, and its owner's allowance, for as long as it runs. It belongs under a service manager.

Quick reference

Find parked jobs:

btjlist -HR -F "%N %U %h %P %W %r %x"

Make a parked job runnable again:

btjchange -N <jobnumber>

Read the two system variables:

btvar CLOAD
btvar LOADLEVEL

Read the three per-user limits:

btulist -H -F "%u %x %t %s"

Raise a user's total load level:

btuchange -T <n> <user>

List one user's running jobs with their load levels:

btjlist -HR -u <user> -F "%N %h %P %L %i"

Test one job's progress state in a script:

btjstat -s Err,Abrt,Canc <jobnumber>

Bottom line: when jobs will not start and CLOAD is well below LOADLEVEL, look at the progress column before anything else, then at the owner's total load level. Both are invisible to the system-wide figures, and neither is reported when it stops a job.