Observations include - jobs start late, start in an unpredictable order, or fail to run at all, whilst every system-wide indicator appears healthy.
Symptoms typically reported:
- Mission-critical jobs start minutes or hours after their scheduled time
- The same job starts at a different time each day for no apparent reason
- Daily jobs occasionally do not run at all
- Forcing a job with f or g does not start it immediately
- CLOAD is well below LOADLEVEL, so there is apparently spare capacity
- Increasing LOADLEVEL, STARTLIM or STARTWAIT makes no difference
The cause is usually that a user's total load level has been consumed by jobs the scheduler still believes are running.
Two limits to check
Xi-Batch applies two independent load constraints before starting any job. Both must be satisfied.
System-wide limit : The job's load level is added to CLOAD. If the total would exceed LOADLEVEL, the job does not start. This limit is visible to everyone through btvlist.
Per-user limit : The scheduler separately sums the load levels of all jobs currently running for the owning user and compares that total against the user's Totll value in the administration file. If the job would take that user over Totll, the job does not start - regardless of how much system-wide headroom exists. This sometimes catches people out.
A third limit, Maxll, caps the load level of any single job the user may submit. It is rarely the cause of a start delay but is worth checking at the same time.
Why the system-wide view looks healthy
Consider a site with LOADLEVEL at 40000 and a CLOAD reading of 18000. That is 45 per cent utilisation and there is no reason to suspect a capacity problem.
Now suppose the busiest user has a Totll of 10000, and seven of that user's jobs are stranded in Run state at a load level of 1000 each. That user has 3000 of allowance left - three concurrent jobs - no matter what LOADLEVEL says. Every job that user owns is queueing behind a three-slot bottleneck whilst the system reports ample capacity.
This is why the usual remedies have no effect:
- Raising LOADLEVEL relaxes a limit that was never binding
- Raising STARTLIM releases more jobs per tranche, but they still cannot start
- Raising STARTWAIT changes the retry interval, not the outcome
How load becomes stranded
A job holds its load level from the moment it starts until the scheduler records its completion. If the scheduler never sees the completion, the load is never released.
Common causes:
- The scheduler was stopped or restarted whilst jobs were running
- The host was rebooted without a clean shutdown of the batch system
- A job's process was killed externally, outside the scheduler's control
- A job with network scope was running on a remote host that became unreachable
- A long-running process legitimately never exits, having been submitted as an ordinary job rather than run as a service
The result is a job showing progress Run with no corresponding process. It consumes its share of CLOAD and its owner's Totll indefinitely. Because nothing errors and nothing is logged, these accumulate quietly over months.
Two secondary effects can make the symptoms confusing:
Unpredictable start times : When jobs queue behind a load limit, the scheduler starts the highest-priority job first. If every job on the system carries the same priority - a common outcome when the priority scheme has never been configured - the order in which waiting jobs claim a freed slot is effectively arbitrary. The same job starts at a different time each day.
Jobs that never run : A repeating job that cannot start at its due time is handled according to its reschedule option. Where that option is set to skip, the occurrence is abandoned and the schedule advances. The job does not run and no error is raised.
Diagnosis
Start with the system variables, then move to the per-user picture.
Check the system-wide position:
btvlist CLOAD LOADLEVEL STARTLIM STARTWAIT
If CLOAD is comfortably below LOADLEVEL, the system-wide limit is not your problem. Note the CLOAD figure - you will compare it against genuinely active work shortly.
List everything the scheduler believes is running:
btjlist -F "%N %U %h %P %L %i"
The format codes give job number, user, title, progress, load level and process id. Filter for jobs whose progress is Run, Strt, Init or Fin.
For each one, check whether the process actually exists:
ps -p <pid>
A job in Run state whose process id does not exist is stranded. Its load is unavailable to the system and to its owner, and it will stay that way until the job is cleared.
Read the affected user's limits:
btuser -d
Run as that user this reports their maximum load level, total load level and privileges. An administrator can view the whole table interactively with btuser, where the Totll column shows each user's total.
Now do the arithmetic. Add the load levels of that user's stranded jobs and subtract from their Totll. The remainder is the number of jobs they can actually run concurrently. If that figure matches the concurrency you observe in practice, the diagnosis is confirmed.
Confirming from the job audit trail
Where LOGJOBS is enabled the audit trail settles the question definitively.
Stranded jobs produce no entries at all. If a job number appears in the queue with progress Run but has no started or completed record anywhere in the log, it has been in that state for at least the retention period of the log:
awk -F'|' -v job=12345 '$3 == job' /path/to/xibatch.log
The audit trail also reveals the bottleneck directly. Count how many of a user's jobs are running concurrently at any moment; if that count never exceeds a fixed number over days of operation, that number is the user's remaining headroom rather than a coincidence.
Where a delayed job starts in the same second that another job belonging to the same user completes, it was waiting for a load slot and nothing else.
Clearing stranded jobs
Confirm the process is genuinely absent before touching anything. Resetting a job whose process is still alive can result in two copies running concurrently.
The scheduler regards these jobs as running, so the progress code cannot simply be reset with P in btq. Use the kill or cancel command instead:
- In btq, move the cursor to the job and press K, then select a signal from the [Int Quit Term Kill] prompt
- From the command line, btjchange -C cancels a job by number
Once the job is released the load returns to CLOAD and to the owner's total. The job can then be set runnable again with r, or its progress code reset with P.
Release stranded load in a controlled window rather than all at once. Freeing a large block of load will let every waiting job start simultaneously, which is precisely the surge that STARTLIM exists to prevent. Either clear the jobs in small batches, or lower LOADLEVEL temporarily so that the backlog drains gradually.
When to raise Totll
Clearing stranded jobs restores capacity that was already provisioned. Raising Totll provisions more. Do the first before considering the second, otherwise you are simply raising a ceiling that will be consumed again at the same rate.
If a user genuinely needs more concurrency after cleanup, the total load level is set per user in btuser:
- Move the cursor to the user and press t to set the total load level
- Press m for the maximum load level of any single job
- Press T and M to change the system defaults applied to new users
Changes are saved on exit from the program.
Bear in mind that Totll and Maxll work together. If every command interpreter carries a load level of 1000 and Maxll is also 1000, the load mechanism is doing nothing except counting jobs. Getting real value from it means giving resource-hungry jobs a higher load than trivial ones, which requires Maxll to be raised as well and the special create privilege to set job load levels individually.
Preventing recurrence
Set a maximum elapsed run time : In btq, press u to edit the process parameters for a job, then R to specify the maximum permitted elapsed run time. An over-running job is then killed automatically with the signal set by K, after a grace period set by g. Jobs with a genuine upper bound on their runtime should always have this set - it is the single most effective guard against stranded load.
Review the queue after every scheduler restart : Any job running when the scheduler stops is a candidate for becoming stranded. Make a check of jobs in Run state part of the restart procedure.
Monitor the CLOAD floor, not just the peak : CLOAD at a quiet period should fall close to zero. A floor that never drops below a certain value is stranded load, and the size of that floor tells you how much has accumulated.
Enable the job audit trail : Setting LOGJOBS costs almost nothing and turns this class of investigation from guesswork into arithmetic. Without it, the concurrency ceiling that gives the diagnosis away is invisible.
Do not submit long-running services as batch jobs : Processes intended to run continuously will hold their load level for as long as they run. They belong under a service manager, not in the batch queue.
Quick reference
Check the system-wide limits:
btvlist CLOAD LOADLEVEL STARTLIM STARTWAIT
List jobs with progress, load and process id:
btjlist -F "%N %U %h %P %L %i"
Test whether a running job's process still exists:
ps -p <pid>
Show the current user's load levels and privileges:
btuser -d
Release a stranded job by number:
btjchange -C <jobnumber>
Bottom line: when jobs will not start and CLOAD is well below LOADLEVEL, the limit being hit is almost always the owning user's total load level. Look for jobs stranded in Run state before changing any scheduler parameter - the capacity is usually already there, just unavailable.