Tracking Xi-Batch Job Activity with LOGJOBS

Understanding Job Audit Trails

LOGJOBS is an Xi-Batch system variable, not a setting in the master configuration file. It names the destination for the scheduler's job log. While it holds the empty string - which is how a new installation starts - no job log is written and nothing reports that fact.

The variable lives in the scheduler's saved variable file alongside LOADLEVEL, CLOAD and LOGVARS, so its value survives a normal stop and start: the scheduler reopens the destination as it reloads the variable file. Two situations reset it to empty and silently switch logging off:

A scheduler started with no saved variable file

a new installation, or one whose variable file has been removed or reinitialised. The system variables are recreated with their defaults, and LOGJOBS comes back empty.

A migration to another system

the variable export tool skips every system variable, so LOGJOBS, LOGVARS, LOADLEVEL and CLOAD are not carried by the export. Job logging is off on the new machine until it is set again. See the Xi-Batch migration article, and check LOGJOBS as a routine post-migration step.

What Gets Logged

Job logging records events against a job in the queue. The scheduler writes an entry when:

A job is created

submitted to the queue.

A job starts, or cannot be started

the scheduler begins execution, or the attempt fails.

A job finishes

as completed, error or abort, decided by the job's exit-code ranges (see Status Codes).

A job is cancelled, deleted or killed

by an operator, or automatically when it outlives its run-time limit or its grace period, or when it is auto-deleted after the retention interval.

A job is forced

started immediately, with or without advancing its scheduled time.

A job's ownership, group, mode or details change

including the daylight-saving adjustment applied to scheduled times.

The machine running a job is lost

the scheduler marks the job aborted on behalf of the failed host.

Variable assignments are logged separately, through LOGVARS. Scheduler starts, stops, licence messages and panics go to the scheduler's own report file, btsched_reps, which is a different file with a different format.

Enabling Job Logging

LOGJOBS is set with btvar, which requires a running scheduler. The value is the argument to -s; the variable name is the positional argument, and the order matters.

Log to file:

btvar -s jobaudit.log LOGJOBS

A relative name is resolved against the scheduler's working directory, which is the spool directory - by default /var/spool/xi/batch - so jobaudit.log becomes /var/spool/xi/batch/jobaudit.log. An absolute path is used as given.

Log to a program:

btvar -s '|/usr/local/bin/job-logger.sh' LOGJOBS

A value whose first character is | is treated as a shell command line rather than a file name. See "Logging to Programs" below for what this involves and who may configure it.

Disable logging:

btvar -s '' LOGJOBS

The empty string turns logging off and closes the current destination.

Who may set it. The variable is owned by the Xi-Batch system user, batch, and grants write permission to its owner only. Root and the batch user both hold full Xi-Batch privileges and can set it; other users are refused. Run the command as one of those two accounts.

A new value takes effect immediately: the scheduler closes the previous destination and opens the new one as the assignment is applied. If the destination cannot be opened - a bad path, a directory the batch user cannot write, a command that fails to start - the scheduler makes no report and logging is simply off, so confirm afterwards that entries are arriving.

Log File Permissions

When logging to a file, the scheduler sets the file's owner, group and mode from the LOGJOBS variable each time it opens the file:

File owner and group

the owner and group of the LOGJOBS variable - by default the batch system user and its group.

File mode

built from the variable's read and write permissions alone. Read on the variable gives r on the file, write gives w, for user, group and others independently. The variable's other permission bits take no part, and no execute bit is ever set. The shipped defaults are read and write for the owner and read for the group, giving mode 0640.

Show what is currently in force with:

btvlist -F '%N %U %G %M %V' LOGJOBS

The mode column reads as U:, G: and O: groups, in which R is read and W is write.

To let your own tooling read the log, grant read to the group that tooling runs as - or to others - on the variable:

btvar -M 'G:+R' LOGJOBS

A mode or ownership change does not reach the existing log file on its own. The scheduler applies owner and mode only when it opens the file, and changing the variable's mode does not reopen it. Re-assign the value to make the change effective:

btvar -s jobaudit.log LOGJOBS

Log Entry Format

Each log entry is a single line with pipe-delimited fields:

05/01/2026|10:22:43|13741|date|completed|jmc|users|150|1000

Field order:

1. Date - dd/mm/yyyy, with a four-digit year. In time zones four or more hours behind UTC the first two fields are exchanged, giving mm/dd/yyyy
2. Time - HH:MM:SS, local time
3. Job Number - the job number, or machine:jobnumber for a job belonging to another host
4. Job Title - the job's title, or <Unnamed job> if it has none
5. Status Code - the event (see below), prefixed with machine: when the request came from a remote host
6. User - job owner user name
7. Group - job group name
8. Priority - job priority value
9. Load Level - job load level value

There is no separator after the load level; the line ends there. Each entry is flushed as it is written, so a file sink is up to date as soon as the event happens.

Status Codes

The status field is one of the following. They are written in lower case, and several contain a space, which matters when writing a parser. The strings are taken from the installed help file, so a site with a customised or translated help file may see different words.

abort

the job finished with a status outside both its exit-code ranges, or was killed by a signal

auto delete

the job was removed automatically after its retention interval expired

cancel

a queued job was cancelled before it started

chgrp

the job's group was changed

chmod

the job's permissions were changed

chown

the job's owner was changed

completed

the job finished with a status in its normal exit-code range

create

the job was submitted to the queue

delete

the job was deleted from the queue

dst adjust

the job's next start time was moved by a daylight-saving adjustment

error

the job finished with a status in its error exit-code range, or the scheduler could not start it at all

exceeded grace period

the job was still running after its grace period and was killed outright

exceeded runtime

the job passed its run-time limit and was signalled

force-run

the job was forced to run without advancing its scheduled time

force-start

the job was forced to start immediately

jdetails

other job details were modified (conditions, assignments, times and so on)

manual kill

a running or starting job was killed on request

network-aborted

the machine the job was running on was lost, and the job was marked aborted

started

job execution began

Which of completed, error and abort a job gets is a property of the job, not a fixed rule about exit codes. Each job carries a normal range and an error range, set with btr -X. The shipped defaults are normal 0:0 and error 1:255, so by default exit 0 gives completed and any other exit status gives error; a job killed by a signal gives abort whatever the ranges are.

Processing Log Files

The pipe-delimited format is straightforward to parse. Match the status field case-sensitively in lower case, and allow for the machine: prefix on entries driven from a remote host.

Count jobs by status:

awk -F'|' '{print $5}' /var/spool/xi/batch/jobaudit.log | sort | uniq -c

Find failed jobs, local and remote:

awk -F'|' '$5 ~ /(^|:)error$/' /var/spool/xi/batch/jobaudit.log

Track a specific user's jobs:

awk -F'|' '$6 == "jsmith"' /var/spool/xi/batch/jobaudit.log

Daily job completion count:

awk -F'|' '$5 == "completed" {print $1}' jobaudit.log | sort | uniq -c

Jobs with high load levels:

awk -F'|' '$9 > 5000' /var/spool/xi/batch/jobaudit.log

Logging to Programs

A LOGJOBS value beginning with | is a shell command line. The scheduler starts it through the shell and writes each log line to its standard input.

Four properties govern how such a sink must be written and who may install one.

Only a local administrator may set it. The scheduler refuses a | value that arrives from a network peer or from a job's own variable assignment, and refuses it before storing it, so the value cannot be left behind to be opened at the next start. A sink is configured on the machine, by root or the batch user, with btvar. Cluster-wide configuration by broadcasting the variable does not work and is not intended to.

It runs with the scheduler's privileges. The command is run by the scheduler process, not by the job's owner and not by the user who set the variable. Treat the script as system software: own it by root, keep it out of directories other users can write, and give the absolute path - the command inherits the scheduler's environment, whose PATH is that of the service that started it rather than an administrator's login.

The program is restarted periodically. When the scheduler next rewrites its queue files - at most every 300 seconds while there is activity - it closes the pipe and starts the command again. A sink that reads its input in a loop will therefore see end-of-file and be re-run several times an hour. Write it so that repeated invocation is harmless: append rather than truncate, and do any per-run setup idempotently.

A sink that stops reading loses entries silently. If the program exits, the scheduler is not told; entries are discarded until the next queue rewrite starts a fresh copy. If the program reads slowly enough for the pipe to fill, the scheduler blocks writing to it, and it waits for the program to finish when it closes the pipe. Keep the sink fast and non-blocking; do no network calls in it.

Example logger script:

#!/bin/sh
# /usr/local/bin/job-logger.sh
# Runs from the scheduler. Keep it fast; it is re-run periodically.

while IFS='|' read -r date time jobnum title status user group priority load; do
    # Strip any machine: prefix from the status field
    event=${status##*:}

    # Note high-priority failures in syslog
    if [ "$event" = "error" ] && [ "$priority" -gt 150 ]; then
        logger -t xibatch "Job failed: $title (job $jobnum, user $user)"
    fi

    # Append every entry to a file of our own
    printf '%s|%s|%s|%s|%s|%s|%s|%s|%s\n' \
        "$date" "$time" "$jobnum" "$title" "$status" \
        "$user" "$group" "$priority" "$load" \
        >> /var/log/xibatch-jobs.log
done

Use Cases

Compliance auditing

Track who ran what jobs when for regulatory requirements

Performance monitoring

Identify frequently failing jobs or resource-intensive jobs

Capacity planning

Analyse job patterns to optimise load levels and scheduling

Troubleshooting

Review job history when investigating scheduling issues

Billing/chargebacks

Track resource usage by user or group

Security monitoring

Detect unusual job submission patterns

Log Rotation

Job logs grow without limit. Xi-Batch never rotates the log, and it never reopens it. The scheduler holds the file open from the moment the value is set until the value changes or the scheduler stops, and it ignores SIGHUP, so there is no signal that will make it pick up a new file.

That makes the usual rotation recipe destructive: renaming the log and creating a new one leaves the scheduler appending to the renamed - or deleted - file, and the new file stays empty. Nothing reports this, and the loss is only noticed when someone reads the log.

Two arrangements work. The first uses logrotate and is therefore Linux only; the second is a shell script and runs on every platform Xi-Batch ships for.

Rotate by copying and truncating. The scheduler opens the file in append mode, so truncation is safe and writing resumes at the start of the empty file:

# /etc/logrotate.d/xibatch-jobs
/var/spool/xi/batch/jobaudit.log {
    weekly
    rotate 52
    compress
    delaycompress
    missingok
    notifempty
    copytruncate
}

Do not add a create stanza here; copytruncate and create are alternatives, and create is the one that loses entries. A small number of entries written between the copy and the truncation can be lost, which is inherent to this method.

Rotate by moving the file and re-assigning the variable. Re-assigning LOGJOBS closes the old file and opens a new one with the correct owner and mode, so nothing is lost:

#!/bin/sh
# Run as root or as the batch user
LOG=/var/spool/xi/batch/jobaudit.log
DATE=$(date +%Y%m%d)

if [ -f "$LOG" ]; then
    mv "$LOG" "${LOG}.${DATE}"
    btvar -s jobaudit.log LOGJOBS
    gzip "${LOG}.${DATE}"
fi

Integration with Monitoring Systems

Forward job events to an external system from a separate process, reading the log file. Do not make network calls from a LOGJOBS sink: the sink runs inside the scheduler's logging path, where a slow or unreachable endpoint stalls job operations, and where the program is restarted at every queue rewrite.

The dependable arrangement is to log to a file, rotate it as above, and have an independent agent tail or poll that file and forward what it finds. Where a sink is genuinely wanted, keep it to a local, non-blocking hand-off - appending to a spool file or writing to the local syslog daemon - and forward from there.

Verifying Logging is Active

Check the LOGJOBS value:

btvar LOGJOBS

btvar with no operation options prints the variable's current value; empty output means logging is off.

Test logging:

# Submit test job
echo "echo test" | btr -h "Test logging"

# Check log file
tail /var/spool/xi/batch/jobaudit.log

You should see create, started and completed entries for the job.

Troubleshooting

No log entries appearing

Check the value with btvar; an empty value means logging is off, and a fresh variable file or a migration is the usual reason. If the value is set, confirm the batch user can create and write the file in the spool directory. A destination the scheduler cannot open produces no error message anywhere - logging is simply inactive.

Log program not receiving data

Give the command as an absolute path; the scheduler's PATH is not an administrator's. Confirm the program is executable by the batch user and that it does not exit on end-of-file expecting to run once - it is restarted at every queue rewrite. Failure to start the command is not reported in btsched_reps or anywhere else.

Log file grows too large

Rotate with copytruncate, or move the file and re-assign LOGJOBS. Do not rotate with a plain create.

Entries stopped after a rotation

The scheduler is still writing to the rotated file. Re-assign LOGJOBS to reopen the current path.

Permission denied reading the log

The file's owner, group and mode come from the LOGJOBS variable. Grant read on the variable with btvar -M, then re-assign the value so the change reaches the file.