Diagnosing Why a Ready Xi-Batch Job Will Not Start

Understanding Job Ready States

The scheduler walks the whole job queue, in working-priority order, every time anything happens to a job or a variable. For each job it applies a fixed sequence of tests, and the job is started only when every one of them passes. The first test is the job's progress code, shown in the Progress column of btjlist: a job is a candidate for starting only while that column is blank.

This is the single most common reason a job that looks due never starts, because a repeating job whose last run exited with an error stays in the Err state permanently. Its next run time is advanced as usual, so the time column keeps moving and the job looks healthy, but the progress code is never reset and the scheduler passes over it on every scan. The same is true of Abrt and Canc. Only btjchange -N (or the equivalent action in btq) puts the job back into the blank state.

Once the progress code is blank, the remaining tests decide whether the job runs now.

Common Reasons Jobs Wait

These are listed in the order the scheduler applies them. The scheduler writes nothing anywhere when it declines to start a job - the job log records starts, completions and errors, and btsched_reps in the spool directory receives only fatal scheduler errors. So the diagnosis is always made by inspecting the job and the variables, never by reading a log.

Progress code is not blank

The job shows Done, Err, Abrt, Canc, Init, Strt, Run or Fin. Visible in the Progress column.

System load level

The job's own load level added to CLOAD would exceed LOADLEVEL. The scheduler skips this job and carries on down the queue, so a smaller job can still start.

Held on an unavailable remote variable, or already proposed elsewhere

The job names a variable on a host this scheduler has yet to hear from, or another host has offered to run it and the offer is outstanding. Neither state is shown by any command.

Another host's job that is not remote-runnable

A job belonging to a remote host is considered here only if its owner marked it runnable elsewhere. The Export column shows Rem-runnable when it is.

Another host's job whose command interpreter is undefined here

The interpreter named on the job must exist in this machine's command interpreter list. Compare the Cmd Interp column with btcilist.

Run time still in the future

The job runs on the minute boundary, and the scheduler will start it up to three seconds early rather than wait for another pass.

Run time missed, and the job is set to skip

A job that missed its slot by more than a tenth of its repeat interval and is set to skip when it cannot run is moved to the back of the queue and its time advanced to the next occurrence. See the time-settings article.

User's total load level

The job's load level added to the load levels of that user's jobs already running on this machine would exceed the user's Totll limit.

A condition is unsatisfied

One of the job's conditions evaluates false; or the variable it names is undefined or unreadable, which blocks the job whatever the comparison operator; or the variable lives on a host that is offline and the condition is marked critical. See the conditions and assignments article.

Start rate limit reached

The number of jobs already in the process of starting has reached STARTLIM. The scheduler abandons the rest of the scan at that point, so every lower-priority job waits with it. See the article on the job start rate.

The scan stopped before reaching the job

After starting a local job the scheduler stops scanning if CLOAD has reached LOADLEVEL. Jobs further down the queue are left untried until the next scan.

Diagnostic Approach

Step 1: Identify Waiting Job

# List all jobs with the fields that matter here
btjlist -H -F "%N %U %L %p %t %P"

# List just the job numbers of jobs in the waiting state
btjlist -N -F "%N|%P" | awk -F'|' '$2 == "" { print $1 }'

The Progress field is the last piece of the format, and the last piece is never padded, so it compares cleanly against the empty string. Note the job number of the waiting job. Job numbers are zero-padded to a fixed width, and a remote job appears as host:number.

Step 2: Check Job Details

# Everything relevant to the start decision, for one job
btjlist -H -F "%N %U %I %p %L %T %C %e %P" <job_number>

The fields are:

• %N job number, %U owner
• %I command interpreter
• %p priority, %L load level
• %T next run date and time in full
• %C conditions in full, as nameoperatorvalue, comma separated
• %e Export or Rem-runnable
• %P progress code

btjstat tests a job's state rather than reporting it: it prints nothing and returns 0 if the job is in the state asked for and 1 if it is not. Use it in scripts, not for inspection.

# True (exit 0) if the job is waiting rather than running
btjstat -s "" <job_number>

Step 3: Check Load Levels

# View the system load variables, one at a time
btvar LOADLEVEL
btvar CLOAD

# The job's own load level
btjlist -N -F "%L" <job_number>

Calculation:

If CLOAD + job_load > LOADLEVEL, the scheduler skips this job and moves on to the next in the queue.

Example:

LOADLEVEL: 20000
CLOAD: 18500
Job load: 2000

18500 + 2000 = 20500 > 20000  ← Job skipped

Solution:

Wait for running jobs to complete, which reduces CLOAD, or raise LOADLEVEL. The value comes first and the variable name last:

btvar -s 25000 LOADLEVEL

CLOAD is maintained by the scheduler and is read-only. The load-levels article covers them in full.

Step 4: Examine Conditions

# The job's conditions, in full
btjlist -N -F "%C" <job_number>

Each condition prints as the variable name, the comparison operator and the value, with conditions separated by commas. A variable on another host carries a host: prefix. For example:

STATUS=Ready,server2:BACKUP_DONE=Yes

Check the variable's actual value:

btvar STATUS

If STATUS holds "Pending" rather than "Ready", the condition is unsatisfied and the job waits.

Step 5: Check Variable Values

For each condition on the job, verify the variable's value. btvar prints the bare value on one line and nothing else, so it is the one to use in scripts; btvlist lists variables in columns.

# Check one variable
btvar <variable_name>

# List variables with their values and comments
btvlist -H

Three answers block a job:

• The value fails the comparison.
• The variable is undefined, or you may not read it. btvar reports "Unknown variable" and exits 20. An undefined variable blocks the job for every comparison operator, including !=.
• The variable belongs to a host that is offline, and the condition is marked critical.

For remote variables:

# Remote variable format: machine:varname
btvar remotemachine:STATUS

An unrecognised host name gives "Unknown host name". Where the host is known but offline and the condition is critical, the job is blocked; where the condition is not critical, the scheduler ignores that condition and carries on.

Step 6: Verify User Load Limits

Each Xi-Batch user account carries three load-level limits, held in the user file and edited with btuchange or the btuser screen:

Maxll

The largest load level any one of that user's jobs may carry. Enforced when the job is submitted or changed, so it blocks submission rather than starting.

Totll

The largest total load level the user's jobs running on this machine may reach at once. This is the one that holds jobs back.

Specll

The load level given to jobs whose command interpreter has no load level of its own.

Reading them needs administrative read privilege:

# The user's limits
btulist -H -F "%u %x %t %s" <username>

The columns print as User, Maxll, Totll and Specll. The shipped defaults are 1000, 10000 and 1000.

The scheduler compares the waiting job's load level against the sum of the load levels of that user's jobs on this machine whose progress is Init, Strt, Run or Fin:

# Total load the user is already accounted for
btjlist -N -u <username> -F "%L %P" | awk '$2 ~ /^(Init|Strt|Run|Fin)$/ { s += $1 } END { print s+0 }'

The article on jobs waiting while CLOAD is below LOADLEVEL covers the per-user total in depth.

Resolving Common Blocking Scenarios

Scenario 1: Progress Code Not Blank

Symptom:

btjlist -H -F "%N %t %r %P" 0000123
# Jobno   Time         Repeat  Progress
# 0000123 14:00        1 hour  Err

The job ran, exited with an error code, and has stayed in the Err state ever since. Its time keeps advancing, so the schedule looks correct.

Solution:

Fix whatever the job reported, then return it to the waiting state:

btjchange -N 0000123

The same command clears Abrt and Canc. A job that ran once and was retained shows Done, and the same command re-arms it.

Scenario 2: Load Level Exceeded

Symptom:

btvar LOADLEVEL
# 20000
btvar CLOAD
# 19500
btjlist -N -F "%L" 0000123
# 1000
# 19500 + 1000 > 20000 - Job skipped

Solutions:

Option A: Increase LOADLEVEL

btvar -s 25000 LOADLEVEL

Option B: Wait for jobs to complete

Monitor CLOAD:

while true
do
    btvar CLOAD
    sleep 5
done

When CLOAD drops to 19000 or below, the job can start.

Option C: Reduce the job's load level

btjchange -l 500 0000123

Changing a job's load level away from its current value requires the special-create privilege, and the new value must be between 1 and 32767.

Scenario 3: Condition Not Satisfied

Symptom:

btjlist -N -F "%C" 0000123
# STATUS=Ready
btvar STATUS
# Pending

Solutions:

Option A: Set the variable to the required value

btvar -s Ready STATUS

The scheduler rescans as soon as the variable changes, so the job starts at once if nothing else blocks it.

Option B: Remove the condition

Where the condition no longer applies, -y deletes every condition on the job. To keep the others, delete them all and re-state the ones you want:

btjchange -y 0000123
btjchange -y -c 'OTHERVAR=1' 0000123

Option C: Wait for another job to set the variable

%s lists the variables a job assigns to, and %S the assignments in full, so the job that sets a variable can be found by listing them for every job:

btjlist -N -F "%N|%s" | grep STATUS

Scenario 4: Remote Variable Unavailable

Symptom:

btjlist -N -F "%C" 0000123
# server2:BACKUP_DONE=Yes
btvar server2:BACKUP_DONE
# btvar: Unknown variable server2:BACKUP_DONE

When a host goes offline, those of its variables that other hosts' jobs depend on are kept as placeholders with no readable value, and jobs with a critical condition on one of them are held.

Solutions:

Option A: Restore the connection

Ask this scheduler to raise the connection again:

btconn server2

Where that fails, the problem is below Xi-Batch: check that the network path to the host is open and that the scheduler and network daemon are running there.

Option B: Make the condition non-critical

A non-critical condition on an unreachable variable is ignored rather than blocking the job. The -K option applies to the conditions named after it, so the condition has to be re-stated:

btjchange -y -K -c 'server2:BACKUP_DONE=Yes' 0000123

Option C: Use a local variable instead

# Create a local variable (-C creates, -s takes the value, the name comes last)
btvar -C -s Yes BACKUP_DONE

# Point the job at it
btjchange -y -c 'BACKUP_DONE=Yes' 0000123

Scenario 5: User Load Limit Exceeded

Symptom:

btulist -H -F "%u %x %t %s" jsmith
# User    Maxll Totll Specll
# jsmith   1000 10000   1000

btjlist -N -u jsmith -F "%L %P" | awk '$2 ~ /^(Init|Strt|Run|Fin)$/ { s += $1 } END { print s+0 }'
# 9500
# New job load 1000: 9500 + 1000 > 10000 - Job skipped

Solutions:

Option A: Wait for the user's jobs to complete

while true
do
    btjlist -N -u jsmith -F "%N %L %P"
    sleep 10
done

Option B: Increase the user's total (requires administrative write privilege)

btuchange -T 20000 jsmith

The scheduler re-reads the user file when it changes, so no restart is needed; the new limit applies on the next scheduling pass.

Option C: Reduce the job's load level

btjchange -l 200 0000123

Scenario 6: The Scan Never Reached the Job

Symptom:

A job with nothing wrong with it waits while others start. Two mechanisms produce this, and both are ordinary behaviour:

• After starting a job the scheduler stops scanning altogether once CLOAD has reached LOADLEVEL, so jobs below that point in the queue are left untried until the next scan.
• The scan also stops when the number of jobs already starting reaches STARTLIM, whatever the load level.

Queue position follows the working priority, which starts at the job's priority and is adjusted as the job is placed relative to its neighbours.

Solutions:

Option A: Raise the job's priority

btjchange -p 200 0000123

The priority must be between 1 and 255 and within the range the user's account permits.

Option B: Wait for the higher priority jobs to complete

This is deliberate priority ordering working as intended.

Option C: Lower the priority of jobs you own that are ahead of it

btjchange -p 50 <other_job_number>

Systematic Troubleshooting Checklist

Use this checklist to diagnose waiting jobs, in the order the scheduler applies the tests:

1. Progress Check

btjlist -N -F "%N|%P" <job_number>

• Progress field is blank

2. Load Level Check

btvar LOADLEVEL
btvar CLOAD
btjlist -N -F "%L" <job_number>
# Calculate: CLOAD + job_load vs LOADLEVEL

• Load levels allow the job to start

3. Time Check

btjlist -N -F "%T" <job_number>
# Compare against current time

• Run time is in the past

4. Conditions Check

btjlist -N -F "%C" <job_number>

For each condition:

• Variable exists and is readable
• Variable value satisfies the condition
• Remote variables reachable, where the condition is critical

5. User Limits Check

btulist -H -F "%u %x %t %s" <username>
btjlist -N -u <username> -F "%L %P" | awk '$2 ~ /^(Init|Strt|Run|Fin)$/ { s += $1 } END { print s+0 }'

• The user's Totll allows the job

6. Remote Job Check

btjlist -N -F "%N %O %I %e" <job_number>
btcilist

• A job owned by another host is marked Rem-runnable
• Its command interpreter is defined on this machine

Advanced Diagnostics

Trace Variable Changes

If the condition involves a variable that should be changing, LOGVARS records every change. The value is the file name, which is taken relative to the spool directory, and the variable name comes last:

# Enable variable logging
btvar -s varlog LOGVARS

# Watch variable changes
tail -f /var/spool/xi/batch/varlog | grep <variable_name>

Monitor Job Transitions

LOGJOBS records job events in the same way. It records starts, completions, errors and deletions - the scheduler logs nothing when it declines to start a job, so an absence of entries is what a blocked job looks like:

# Enable job logging
btvar -s joblog LOGJOBS

# Watch job state changes
tail -f /var/spool/xi/batch/joblog | grep <job_number>

The job-logging and variable-logging articles cover the two logs.

Check Network Variables

For jobs with remote conditions, the conditions field carries the host prefix, so the remote variables can be read off it and tested one at a time:

# The job's conditions, including any host: prefixes
btjlist -N -F "%C" <job_number>

# Test each remote variable
btvar server1:VAR1
btvar server2:VAR2

Verification After Resolution

After making changes, verify that the job starts:

btjlist -N -F "%N %t %P" <job_number>

The Progress field should move to Init, then Strt, then Run. The scheduler rescans as soon as a variable or a job changes, so the transition follows the change closely. If the job is still waiting, work down the checklist again.

Best Practices

Watch for jobs stuck in Err

A repeating job that fails once stays in Err until someone resets it. Review the progress column regularly.

Use informative variable names

Makes condition troubleshooting easier

Document job dependencies

Note which variables jobs depend on

Monitor variable changes

Enable LOGVARS for critical workflows

Set reasonable load levels

Keep individual job load levels well inside the user's Totll

Use appropriate priorities

Reserve high priorities for truly critical jobs

Test conditions before deployment

Verify conditions work as expected

Provide fallback mechanisms

Use non-critical remote conditions when appropriate