Controlling Xi-Batch Job Flow with Conditions and Assignments

Understanding Conditions

A condition is a test on a variable that must hold before the job may start. A job may carry up to 10 conditions, and all of them must hold at the same moment.

Conditions and assignments both work on Xi-Batch variables. What a variable is, how it is created and how it is permissioned are covered in Working with Variables in Xi-Batch; this article covers only how a job waits on one and how a job changes one.

Condition syntax:

[host:]variable_name operator value

No spaces are allowed anywhere inside it. The parser reads the variable name, then expects the operator immediately, then takes everything remaining as the value. A space before the operator is rejected outright - btr stops with "Bad condition" and the offending argument - and a space after it becomes the first character of the value. Only leading whitespace, before the variable name, is discarded.

# Correct
btr -c 'backup_status=Complete' post-backup.sh

# Rejected - btr: Bad condition 'backup_status = Complete'
btr -c 'backup_status = Complete' post-backup.sh

Operators:

• = or == equal to
• != not equal to
• < less than
• <= less than or equal
• > greater than
• >= greater than or equal

The left-hand side must be a variable name, optionally prefixed with a host name and a colon. It may be up to 19 characters and must begin with a letter or an underscore.

The right-hand side must be a literal value. It cannot be another variable, an expression, a command or an environment reference - there is no way to compare two variables with each other. A value that begins with a digit or a minus sign is taken as a whole number; anything else is text, up to 49 characters. Prefix the value with a colon to force a digit-leading value to be treated as text.

# Compare against the text "0012", not the number 12
btr -c 'partno=:0012' job.sh

Which comparison is used is decided by the value in the condition, not by the variable. If the condition's value is a number, the variable is read as a number and a text variable counts as zero. If the condition's value is text, the variable is converted to text and the two are compared character by character - so 10 sorts before 9. Write numeric conditions against variables you keep numeric.

Simple Condition Examples

# Wait for a backup to finish
btr -c 'backup_status=Complete' post-backup.sh

# Wait for a counter to reach a threshold
btr -c 'processed_count>=100' summary.sh

# Wait for a status to be anything but Pending
btr -c 'status!=Pending' process.sh

# Numeric comparison
btr -c 'error_count<5' continue.sh

A value containing spaces is legal - it is the space before the operator that is not:

btr -c 'extract_status=Not Started' job.sh

Multiple Conditions

Up to 10 conditions, all of which must hold. There is no OR, and there is no grouping.

btr -c 'backup_status=Complete' \
    -c 'validation_status=Passed' \
    -c 'error_count=0' \
    final-report.sh

An eleventh condition is rejected as the command line is read, with "Sorry, you cannot have that many conditions". A ninth assignment is rejected in the same way.

What Happens When a Condition Is Not Met

The scheduler tests a job's conditions each time it reconsiders the queue, immediately before it would start the job. If any condition does not hold, the job is passed over and the scheduler moves on to the next one. Nothing is written to the job log, nothing is written to the scheduler's report file, the job's status does not change, and the job's next run time is not moved.

Two consequences follow, and both cause support calls:

• A job whose condition can never become true waits for ever, in silence. There is no timeout, no expiry and no warning. A job waiting on a misspelt variable name, or on a value that nothing ever sets, sits in the queue looking exactly like a job that is merely waiting its turn.
• A condition on a variable that does not exist, or that the job's owner cannot read, holds the job for every operator - including != and <. An unreadable variable is not treated as "not equal"; it is treated as unknown, and unknown blocks. So deleting a variable a job depends on, or removing the owner's read permission on it, stops the job silently.

Finding out why a job is waiting. Print the job's conditions and then read each variable:

btjlist -H -F '%N %H %C'
btvar backup_status

If the variable name in the listing is one you do not recognise, or reading it gives "Variable does not exist", that is the answer. The article Diagnosing Why a Ready Xi-Batch Job Will Not Start covers the full set of tests the scheduler applies, of which conditions are one.

Remote Variable Conditions

A condition may name a variable on another host by prefixing it with the host name and a colon. The variable must be exported by the host that owns it.

btr -c 'server1:backup_status=Complete' local-job.sh

btr -c 'server1:data_ready=Yes' \
    -c 'server2:data_ready=Yes' \
    merge-data.sh

Critical and Non-Critical Conditions

Criticality decides what happens to a remote condition when the host that owns the variable becomes unreachable. It applies only to conditions that name another host. The flag is accepted on a condition without a host prefix and then discarded, so marking a local condition critical does nothing.

Non-critical is the default. When the owning host goes away, its variables on this machine are discarded or reduced to unreadable placeholders, and a non-critical condition on such a placeholder is skipped entirely - not treated as unmet, not treated as unknown. It ceases to exist for as long as the host is down, and the job runs. Nothing is logged when this happens, so a job that ran without its precondition is indistinguishable afterwards from one that ran with it.

Critical is the opposite: the job is held until the variable comes back. It is held indefinitely and with no message - no timeout, no report-file entry, no change of status.

Neither behaviour is a safe default for every case, so choose deliberately. Mark a condition critical with -k before it, and back to non-critical with -K:

# Critical: hold the job if server1 is unreachable
btr -k -c 'server1:backup_status=Complete' post-backup.sh

# Explicitly non-critical
btr -K -c 'server1:optional_check=OK' process.sh

-k and -K apply to every -c that follows them on the command line, so a mixture is written by switching between them.

Use critical for:

• Dependencies where running without the precondition would do damage
• Any condition whose whole purpose is to protect the job

Use non-critical for:

• Optional checks that should not stop work when a peer is down
• Monitoring and reporting jobs

In btq, on the conditions screen, c toggles the critical mark for remote conditions, and the assignments screen uses the same key.

The critical mark is not shown by btjlist. Neither the condition format nor the assignment format prints it, so btq's condition and assignment screens are the only way to see which way an existing job is set.

Understanding Assignments

An assignment changes a variable's value when the job reaches a given point in its life. A job may carry up to 8 assignments in total, across all stages - not eight at the start and eight more at the end.

An assignment is given with -s, and its syntax is the same shape as a condition and has the same whitespace rule:

[host:]variable_name operator value

Assignment operators:

=

Assign the value

+= or +

Add to a numeric variable

-= or -

Subtract from a numeric variable

*= or *

Multiply a numeric variable

/= or /

Divide a numeric variable

%= or %

Remainder of a numeric variable

The arithmetic operators are applied by the scheduler and are silently skipped in three cases: when the variable holds text, when the value in the assignment is text, and when dividing or taking a remainder by zero. Nothing is logged and the job is unaffected.

The right-hand side is a literal value on the same terms as a condition: numeric if it begins with a digit or a minus sign, text otherwise, up to 49 characters, with a leading colon to force text.

Assignment Stage Flags

Which stage an assignment applies to is set by a separate option, -f, taking a string of letters. -f applies to every -s that follows it on the command line and stays in force until the next -f, so the order of the options matters.

S (Start)

Apply when the job starts

N (Normal)

Apply when the job finishes with an exit code in its normal range

E (Error)

Apply when the job finishes with an exit code in its error range

A (Abort)

Apply when the job is killed by a signal, or exits with a code in neither range

C (Cancel)

Apply when the job is cancelled before it runs

R (Reverse)

Apply the opposite operation at the end of the job as well

The normal and error ranges are the job's own. By default the normal range is exit code 0 only and the error range is 1 to 255, so N is "exited 0", E is "exited non-zero" and A is "killed by a signal". Both ranges can be changed per job with -X, in which case N, E and A follow the ranges rather than the defaults.

If no -f is given, the flags are SNEAR - the assignment is applied at start, at every one of the three end states, and in reverse. That is rarely what is wanted for a plain status update, so give -f explicitly.

The letters differ between the command line and btq: on the btq assignments screen the normal-exit flag is O (Ok), not N, and the others are S, E, A, C and R as here.

Simple Assignment Examples

Update a status at start and at each outcome:

btr -f S  -s 'status=Running' \
    -f N  -s 'status=Complete' \
    -f EA -s 'status=Failed' \
    job.sh

Count the runs:

btr -f S -s 'job_count+=1' job.sh

Read the assignments back:

btjlist -H -F '%N %H %S'

The display prints the flag letters, a colon, then the assignment - for example SN:status=Running.

The Reverse Flag

An assignment carrying R is applied twice: once at the stage or stages named, and again in reverse at the end of the job. The reverse of an arithmetic operator is its inverse - += becomes -=, *= becomes /= and so on. The reverse of a plain assignment is not the previous value. It sets the variable to zero if the value was numeric, and to the empty string if it was text.

# Set the flag to 1 at start; zero it at every end state
btr -f SNEAR -s 'in_progress=1' critical.sh

That covers normal, error and abort in one assignment. It does not cover cancellation unless C is added, and it does not run at all if the job never reaches an end state - see Mutual Exclusion below.

Capturing the Exit Code or Signal

Two literal values are recognised on the right-hand side of a plain assignment: exitcode and signal.

btr -s 'last_exit=exitcode' job.sh
btr -s 'kill_signal=signal' job.sh

Both must be used with = and with no other operator. Both ignore any -f given: they are applied at whichever of the normal, error and abort ends the job actually reaches, and never at start or cancel. The variable receives a whole number.

Changing Conditions and Assignments on an Existing Job

btjchange takes the same -c, -s, -f, -k, -K, -b and -B options as btr. It is not interactive: with no change options at all it prints "You didn't specify any changes to the jobs" and stops without looking at the job numbers. The interactive editor is btq.

btjchange adds to the existing lists; it does not replace them. Clear them first:

# Replace the whole condition list
btjchange -y -c 'backup_status=Complete' 87001

# Replace the whole assignment list
btjchange -z -f S -s 'status=Running' -f N -s 'status=Done' 87001

-y cancels the conditions, -z the assignments.

Criticality cannot be toggled on its own. -k and -K change a flag that is applied as each condition is parsed, so a btjchange carrying only -k copies the existing conditions unchanged and does nothing. The condition has to be restated:

btjchange -y -k -c 'server1:backup_status=Complete' 87001

Remote Assignments

An assignment may name a variable on another host, which must be exported by its owner. The change is forwarded to the owning host and applied there.

btr -f N -s 'server1:import_complete=Yes' export-data.sh

Remote assignments have their own criticality pair, -b (critical) and -B (not critical), on the same terms as -k and -K for conditions: not critical by default, and only ever applied to an assignment that names a host.

Mutual Exclusion

Two jobs that must never run at the same time - because they need the same tape drive, or update the same database - are a common requirement, and a variable used as a semaphore is the traditional answer. There are two ways to build one and they are not equally sound.

The job-level form is the one older documentation gives: both jobs carry the condition that the lock is free, decrement it at start and increment it at the end.

btvar -C -s 1 LOCK

btr -c 'LOCK>0' -f S -s 'LOCK-=1' -f NEA -s 'LOCK+=1' job-xyz.sh
btr -c 'LOCK>0' -f S -s 'LOCK-=1' -f NEA -s 'LOCK+=1' job-abc.sh

Initialising the variable to a larger number caps the group at that many concurrent jobs rather than one.

What holds. On a single host, for jobs whose conditions and assignments are all on that host's own variables, the scheduler re-tests the conditions after taking the variable lock and immediately before applying the start assignment. Two ordinary local jobs due at the same moment will not normally both see the lock free.

What does not hold, and why this form must not be relied on where being wrong would do damage:

• Across hosts nothing covers the sequence. A job on the host that owns the lock variable and a job on another host are started by two different schedulers, and the check that protects a variable against a simultaneous update applies only to requests that began on the owning machine. Both jobs can see the lock free and both can run, with nothing reported.
• Nothing releases the lock if a job never reaches an end state. Stopping the scheduler kills running jobs outright, so their end-of-job assignments never run and the lock stays taken. Nothing times it out and nothing reports it, so every job waiting on it then waits for ever - see What Happens When a Condition Is Not Met.
• A stage the release was not attached to leaves the lock taken. Use NEA, or the reverse flag, so that error and abort release it as well as a normal finish; and remember that cancellation is not in either set unless C is added.
• The arithmetic can be skipped in silence. If anything ever stores text in the lock variable, the decrement and the increment both stop happening and the condition then reads the variable as zero for ever.

The script-level form is the sound one. btvar can test and set a variable in one atomic operation - the write happens only if nobody has changed the variable since btvar read it. Take the lock inside the job rather than through its conditions:

#!/bin/sh
btvar -s 1 -eq 0 update_lock || { echo "lock held"; exit 1; }
trap 'btvar -s 0 update_lock' EXIT INT TERM
do-the-work

This is guarded by the scheduler, it reports failure to take the lock as an exit status the script can act on, and the trap releases it on paths a job assignment would miss. Its own limits - owning host only, a retry ceiling, and no protection on a conditional delete - are set out in Working with Variables in Xi-Batch.

Where neither form is wanted, express the constraint in the scheduler instead: make the second job conditional on the first job's completion status rather than on a lock, so a missed release leaves one job waiting rather than a stale lock blocking the whole group; or give the jobs load levels that make two of them exceed LOADLEVEL, so the scheduler will not run them together.

Where a counter must be correct, let the scheduler do the arithmetic with += in an assignment rather than reading and writing it from a script with two separate btvar calls, which has an unguarded gap between them.

Workflow Patterns

Sequential Job Chain

btr -f N -s 'step1_done=Yes' step1.sh

btr -c 'step1_done=Yes' -f N -s 'step2_done=Yes' step2.sh

btr -c 'step2_done=Yes' step3.sh

Reset the flags at the start of each cycle, or the second run of the chain starts with every step already marked done.

Fan-Out

btr -f N -s 'parent_done=Yes' parent.sh

btr -c 'parent_done=Yes' child1.sh
btr -c 'parent_done=Yes' child2.sh
btr -c 'parent_done=Yes' child3.sh

Fan-In

btr -f N -s 'input1_ready=Yes' input1.sh
btr -f N -s 'input2_ready=Yes' input2.sh
btr -f N -s 'input3_ready=Yes' input3.sh

btr -c 'input1_ready=Yes' \
    -c 'input2_ready=Yes' \
    -c 'input3_ready=Yes' \
    merge-all.sh

Counter With a Threshold

btvar -C -s 0 file_count

btr -f N -s 'file_count+=1' process-file.sh

btr -c 'file_count>=100' -f S -s 'file_count=0' generate-summary.sh

Keep the counter numeric. If anything stores text in it the additions stop happening, silently, and the summary job then waits for ever.

State Machine

btvar -C -s Initialise workflow_state

btr -c 'workflow_state=Initialise' -f N -s 'workflow_state=Extract'   initialise.sh
btr -c 'workflow_state=Extract'    -f N -s 'workflow_state=Transform' extract.sh
btr -c 'workflow_state=Transform'  -f N -s 'workflow_state=Load'      transform.sh
btr -c 'workflow_state=Load'       -f N -s 'workflow_state=Complete'  load.sh

A state machine driven this way stalls permanently at whichever step fails, because a failing job does not advance the state and nothing reports the stall. Add an error assignment at each step that moves the state somewhere an alert job is waiting on.

Complex Example: Production Workflow

# Variables
btvar -C -s "Not Started" extract_status
btvar -C -s 0 extract_count
btvar -C -s "Not Started" transform_status
btvar -C -s "Not Started" load_status

# Extract job, daily at 01:00
btr -T '01:00' -r Days:1 \
    -f S  -s 'extract_status=Running' -s 'extract_count+=1' \
    -f N  -s 'extract_status=Complete' \
    -f EA -s 'extract_status=Failed' \
    extract-data.sh

# Transform job, waits for extract
btr -c 'extract_status=Complete' \
    -f S  -s 'transform_status=Running' \
    -f N  -s 'transform_status=Complete' -s 'extract_status=Not Started' \
    -f EA -s 'transform_status=Failed' \
    transform-data.sh

# Load job, waits for transform
btr -c 'transform_status=Complete' \
    -f S  -s 'load_status=Running' \
    -f N  -s 'load_status=Complete' -s 'transform_status=Not Started' \
    -f EA -s 'load_status=Failed' \
    load-data.sh

# Alert jobs, one per failure state
btr -c 'extract_status=Failed' \
    -f N -s 'extract_status=Not Started' send-alert.sh

btr -c 'transform_status=Failed' \
    -f N -s 'transform_status=Not Started' send-alert.sh

btr -c 'load_status=Failed' \
    -f N -s 'load_status=Not Started' send-alert.sh

Note -T for the start time. -t is the delete time - a number of hours after which the job is removed if that long has passed since it was queued or last ran - and using it here would set no start time at all.

Each job here uses two -s options after one -f; that is what -f persisting means in practice, and it is why the four -s options in the transform job fall into two pairs.

Best Practices

Quote every -c and -s argument, and put no spaces in it. The single most common mistake in this area is a space before the operator, which is rejected, or a space after it, which silently becomes part of the value.

Give -f explicitly. The default is SNEAR, which applies the assignment at start, at all three end states and in reverse. An assignment that was meant to fire once at completion will otherwise fire four times.

Cover every outcome. A status variable set at start and on normal exit only is left saying Running for ever when the job fails:

# Covers all three end states
-f S  -s 'status=Running' \
-f N  -s 'status=Success' \
-f EA -s 'status=Failed'

Reset status variables at the start of a cycle, not at the end of the previous one - an aborted run leaves the end-of-cycle reset unexecuted.

Choose criticality deliberately on every remote condition. The default lets the job run when the other host is unreachable, and says nothing about it.

Keep numeric variables numeric. Arithmetic assignments on a text variable are skipped in silence, and a numeric condition against a text variable reads it as zero.

Take a lock in the job, not in its conditions. Where two jobs must never overlap, btvar's atomic test-and-set inside the script is the sound route; the condition-and-assignment form is weaker in three specific ways set out above.

Check what you actually submitted. A job's conditions and assignments are worth reading back rather than assumed:

btjlist -H -F '%N %H %C %S'

Split large workflows. The limits are 10 conditions and 8 assignments per job, and a job needing more of either is usually two jobs.