Deleting Xi-Batch Jobs in Bulk

When to Use Bulk Operations

Bulk job removal is useful for:

• Clearing test jobs after development work
• Removing jobs submitted with incorrect parameters
• Queue maintenance during system migration
• Recovering from job submission errors

Read this before you delete anything. A job that runs once and carries no repeat time takes itself out of the queue as soon as it finishes. What accumulates in the queue is therefore the site's timed and repeating work, and deleting one of those jobs is irreversible: the scheduler unlinks the job's SP script file in the spool directory as it removes the queue entry, and that file is the only copy of the job's script. Take a copy first with btjdel -u -k, as shown below, unless you already know the jobs are disposable.

Understanding Force Deletion

btjdel refuses to touch a job that has started, and says so:

btjdel: Job 123456 is running - use "-y" argument to force kill.

The force flag is -y (-Y and --force are the same flag). It changes what happens to those started jobs:

• Every started job named on the command line is signalled: SIGTERM (15) by default, or the signal number given with -K.
• The signal goes to the job's whole process group, so the job script's children are signalled too.
• btjdel then sleeps once for the -S time, which defaults to 4 seconds, and re-reads the queue.
• Jobs that have stopped by then are deleted. Jobs still going are reported as did not die in 4 seconds - try -K9, are left in the queue, and make btjdel exit 1.

Force applies only to jobs that have started; for a waiting job it changes nothing. It leaves the permission checks in place: the scheduler requires delete permission on each job, and kill permission as well to signal a running one, and otherwise answers You are not permitted to perform the requested operation on job .... Those permissions come from the job's own mode bits combined with the user's entry in btufile, so root has no automatic power over other users' jobs beyond the privileges its btufile entry carries (by default root and the batch account are given every privilege, which does let them delete other users' jobs).

Force is a way of stopping work that is under way. Where a job simply refuses to die, raise the wait with -S and then escalate the signal with -K 9.

Bulk Deletion Procedure

btjdel accepts a list of job numbers in one call (btjdel [args] job [job...]), so no loop is needed. Job arguments are plain numbers, optionally prefixed with a host name as host:1234 for a remote job. Ranges, wildcards and patterns are rejected with Job argument ... is not numeric.

# Collect the job numbers of the local jobs you can see, one per line.
# -F "%N" prints only the job number, -N suppresses any header,
# -L excludes jobs belonging to other hosts.
btjlist -N -L -F "%N" > /tmp/job_ids.txt

# Look at what you are about to remove before you remove it
btjlist -N -L -H -F "%N %U %H %P"

# Take a copy of each job first. -u writes a pair of files named
# C and J followed by the job number into the current directory;
# -k keeps the jobs in the queue while you do it.
xargs btjdel -u -k < /tmp/job_ids.txt

# Delete the jobs that are only waiting
xargs btjdel < /tmp/job_ids.txt

# Kill and delete the jobs that had already started, giving them
# 30 seconds to finish rather than the default 4
xargs btjdel -y -S 30 < /tmp/job_ids.txt

# Confirm what is left
btjlist -H

btjdel reports each failure on standard error and carries on with the rest of the list. Its exit status is a poor guide to what happened: a job that was skipped because it was running, and a job refused on permissions, both leave the status at 0. It returns 13 when a job number could not be found and 1 when a signalled job outlived the wait. Check the queue afterwards with btjlist rather than trusting the status.

The -F and -N options matter in a script. btjlist takes its defaults from the BTJLIST environment variable and from the per-user configuration files on the batch configuration path, so the format and the header of a bare btjlist are whatever the site has set. Naming the format explicitly makes the output yours. In the shipped default format, %N %U %H %I %p %L %t %c %P, the job number is indeed the first column, but it is padded with leading zeroes to the width of the longest job number in the list, and remote jobs appear as host:number.

Selective Deletion

btjlist can do most of the selecting itself, which is safer than filtering its text output:

# Jobs belonging to one user, or one group
btjlist -N -L -u fred -F "%N"
btjlist -N -L -g devel -F "%N"

# Jobs in one named job queue
btjlist -N -L -q nightly -F "%N"

To select on state, put a separator in the format string rather than relying on spaces. The progress column is the last field of the default format and has no trailing space, and it is empty for a job that is waiting to run, so a filter such as grep " Run " matches nothing:

# Jobs that are actually executing
btjlist -N -L -F "%N|%P" | awk -F'|' '$2 == "Run" { print $1 }'

The progress values are: empty for a job waiting to run, then Done, Err, Abrt, Canc, Init, Strt, Run and Fin. btjdel treats Init, Strt, Run and Fin alike as "running" and needs -y for all four.

Selecting by job number as a proxy for age gives wrong answers. A job number is the process id of the command that submitted the job, raised by 80000 whenever that number is already in use, so job numbers neither increase over time nor stay within a range. Select on the time columns instead:

# Job number and the full date and time each job is next due
btjlist -N -L -F "%N|%T"

What This Achieves

After a bulk deletion:

• The waiting jobs you listed have gone from the queue, along with their script files
• Jobs that had started have been signalled, and those that stopped within the wait have gone too
• Jobs that outlived the wait are still in the queue and still named in the btjdel output
• Jobs you lacked permission on are still in the queue, each with its own error line
• Copies of anything unqueued with -u are in the directory you ran the command from

Output already produced by a job is handled by the scheduler's own completion path: captured standard output and error are mailed or written to the job's owner when the job ends, then removed. Killing a job with -y therefore forfeits whatever the run would have gone on to produce, while what it had already written is still delivered.

This is an administrative tool for queue maintenance. Where jobs become stuck regularly, treat that as a separate problem and investigate scheduler health and IPC resources.