Tracking Xi-Batch Variable Changes with LOGVARS

Understanding Variable Audit Trails

LOGVARS is an Xi-Batch system variable, not a setting in the master configuration file. It names the destination for the scheduler's variable log, which records changes made to variables in the queue. While it holds the empty string - which is how a new installation starts - no variable log is written and nothing reports that fact.

The variable lives in the scheduler's saved variable file alongside LOADLEVEL, CLOAD and LOGJOBS, so its value survives a normal stop and start. Two situations reset it to empty and silently switch logging off: a scheduler started with no saved variable file - a new installation, or one whose variable file has been removed or reinitialised - and a migration to another system, because the variable export tool skips every system variable. Check LOGVARS as a routine step after either.

LOGVARS is the twin of LOGJOBS, and three things about them are identical: how the log file's owner, group and mode are derived; what is involved in sending the log to a program; and the fact that the product never rotates the log. Those are described in full in the LOGJOBS article, Tracking Xi-Batch Job Activity with LOGJOBS, and are summarised here. The two log formats are different, and a parser written for one will not read the other.

What Gets Logged

Variable logging captures these operations:

Value assignments

a value set by a user command, by the API, or by a job's own assignment list - including the arithmetic forms, which are logged as an assignment showing the resulting value.

Variable creation

a new variable is added.

Variable deletion

a variable is removed.

Ownership and group changes

the variable's owner or group is changed.

Permission changes

the variable's modes are changed.

Comment changes

the variable's comment is changed.

Export and cluster changes

the variable is made exported, clustered or local only.

Variable renaming

the variable is given a new name.

Four things a reader may expect to find are absent from the log, and knowing this saves a long search:

Changes arriving from another host

when an exported variable is changed on the host that owns it and the new value is broadcast to this one, this scheduler applies the value without writing a log entry. Only the owning host records the change. The same applies to a variable deleted on its owning host, and to the variables discarded when a host leaves the network.

Changes you make to another host's variable

a request against machine:variable is forwarded to that machine and logged there, not here.

The current load level

CLOAD is updated directly by the scheduler as jobs start and finish, and is never logged. LOADLEVEL, which an administrator sets, is logged normally.

The values loaded at scheduler start

reloading the saved variable file is not a change and produces no entries.

Job events are logged separately, through LOGJOBS. Scheduler starts, stops, licence messages and panics go to the scheduler's own report file, btsched_reps, which is a different file with a different format.

Enabling Variable Logging

LOGVARS is set with btvar, which requires a running scheduler. The value is the argument to -s; the variable name is the positional argument, and the order matters.

Log to file:

btvar -s varaudit.log LOGVARS

A relative name is resolved against the scheduler's working directory, which is the spool directory - by default /var/spool/xi/batch - so varaudit.log becomes /var/spool/xi/batch/varaudit.log. An absolute path is used as given.

Log to a program:

btvar -s '|/usr/local/bin/var-logger.sh' LOGVARS

A value whose first character is | is treated as a shell command line rather than a file name. See "Logging to Programs" below.

Disable logging:

btvar -s '' LOGVARS

The empty string turns logging off and closes the current destination. The destination is closed before the change is recorded, so switching logging off leaves no entry saying so.

Who may set it. The variable is owned by the Xi-Batch system user, batch, and grants write permission to its owner only. Root and the batch user both hold full Xi-Batch privileges and can set it; other users are refused. Run the command as one of those two accounts.

A new value takes effect immediately: the scheduler closes the previous destination and opens the new one as the assignment is applied. Because the new destination is opened first, the assignment of LOGVARS itself is the first entry in the new log. If the destination cannot be opened - a bad path, a directory the batch user cannot write, a command that fails to start - the scheduler makes no report and logging is simply off, so confirm afterwards that entries are arriving.

Log File Permissions

When logging to a file, the scheduler sets the file's owner, group and mode from the LOGVARS variable each time it opens the file. The owner and group are the variable's; the mode is built from the variable's read and write permissions alone, with no execute bit ever set, which gives 0640 with the shipped defaults. A change to the variable's mode or ownership does not reach an already-open log file until the value is re-assigned. The derivation, the commands to read and change it, and the re-assignment step are the same as for LOGJOBS and are given in full in that article.

Log Entry Format

Each log entry is a single line with pipe-delimited fields:

05/01/2026|09:52:43|cnt|assign|Job start|jmc|users|2011|86742|myjob

Field order:

1. Date - dd/mm/yyyy, with a four-digit year. In time zones four or more hours behind UTC the first two fields are exchanged, giving mm/dd/yyyy
2. Time - HH:MM:SS, local time
3. Variable Name - name of the affected variable. On a rename this is the new name; the old name is not recorded
4. Status Code - what was done to the variable (see below), prefixed with machine: when the request came from a remote host
5. Event Code - the circumstance in which it was done (see below)
6. User - user name the change is attributed to
7. Group - group name the change is attributed to
8. Value - depends on the status code (see below); numeric or string
9. Job Number - present only when the change came from a job
10. Job Title - present only when the change came from a job

A line has eight fields or ten, never nine. Fields 9 and 10 are written as a pair, and only for a change made by a running job; a change made any other way ends after field 8. A parser that assumes a fixed field count will misread half the file.

Field 8 is empty on a delete entry, and the separator before it is still written, so a delete line ends with a trailing pipe. Field 10 is also empty when the job that made the change has no title - unlike the job log, the variable log writes nothing in place of a missing title.

Two fields can contain a pipe character of their own: the value in field 8 and the job title in field 10, both of which are free text a user supplies. Splitting on the first seven separators and treating the remainder as the tail is safer than splitting on every separator, if the site's variable values may contain one.

There is no separator after the last field; the line ends there. Each entry is flushed as it is written, so a file sink is up to date as soon as the change happens.

Status Codes

The status field is one of the following. They are written in lower case except where shown, one contains spaces, and the strings are taken from the installed help file, so a site with a customised or translated help file may see different words.

assign

a value was assigned - by command, by the API, or by a job. Field 8 holds the new value

create

the variable was created. Field 8 holds its initial value

delete

the variable was deleted. Field 8 is empty

rename

the variable was renamed. Field 3 holds the new name and field 8 the value; the old name is not recorded anywhere

chown

the owner was changed

chgrp

the group was changed

chmod

the permissions were changed

chcomment

the comment was changed. Field 8 holds the new comment, not the value

change export flag

the variable was made exported, clustered or local only. Field 8 holds one of Cluster, Exported or Local only, not the value. Note the spaces in both the status code and the third of those words

On a chown, chgrp, chmod or rename entry, field 8 holds the variable's current value, which has not changed. The log records that the ownership or the modes were altered; it does not record what they were altered from or to. For that history, compare against btvlist output taken periodically.

Event Codes

The event code indicates the circumstance of the change:

manual

any request from outside a job - btvar, the btq screens, the graphical clients, or the API. A request that arrived from another host carries that host's name as a machine: prefix on the status code in field 4, while the event code stays manual

Job start

applied by a job as it starts

Job completed

applied by a job that finished in its normal exit-code range

Job error

applied by a job that finished in its error exit-code range

Job abort

applied by a job that was killed by a signal, or finished outside both ranges

Job cancel

applied when a queued job was cancelled before it started

These six are the whole vocabulary of the event field, and note that five of them contain a capital letter and a space.

Understanding Job-Driven Changes

A job carries a list of assignments, each tagged with the stages at which it fires. The stage tags are given with -f and the assignment itself with -s; the letters are S for start, N for normal completion, E for error, A for abort and C for cancel, and they map one for one onto the event codes above; a sixth letter, R, reverses the assignment at the end of the job. The operators are =, +=, -=, *=, /= and %=. Each -f applies to the -s options that follow it. With no -f at all, an assignment fires at job start and is reversed at completion, error and abort, so it produces two log entries per run rather than one.

Example job with assignments:

btr -h "Process data" \
    -f S -s 'cnt+=1' \
    -f N -s 'STATUS=Done' \
    -f E -s 'STATUS=Failed' \
    process-script.sh

Resulting log entries:

06/02/2026|10:15:22|cnt|assign|Job start|jmc|users|2012|87001|Process data
06/02/2026|10:18:45|STATUS|assign|Job completed|jmc|users|Done|87001|Process data

How a job-driven change is attributed. This is the question an audit is usually asking, and the answer is not the obvious one:

The user and group are the job's owner

fields 6 and 7 name the account the job belongs to, not the person who submitted it, changed it or forced it to run. If a job is submitted by one user and reassigned to another, later assignments are attributed to the new owner.

The job is named in fields 9 and 10

the job number and the job title. The title is whatever btr -h gave the job, and is empty if the job has no title - so identify the job by its number and treat the title as a convenience.

A machine: prefix on field 4 names the host that ran the job

not the host that owns the variable and not the host the job was submitted from. A job that ran locally has no prefix.

An assignment against another host's variable is not in this log

it is recorded on the machine that owns the variable.

Processing Log Files

Match the status field case-sensitively, and allow for the machine: prefix on entries driven from a remote host.

Count changes by variable:

awk -F'|' '{print $3}' /var/spool/xi/batch/varaudit.log | sort | uniq -c

Find manual changes:

awk -F'|' '$5 == "manual"' /var/spool/xi/batch/varaudit.log

Track a specific variable:

awk -F'|' '$3 == "STATUS"' /var/spool/xi/batch/varaudit.log

Job-driven assignments:

awk -F'|' '$5 != "manual" && NF == 10' /var/spool/xi/batch/varaudit.log

Variables changed by a specific job:

awk -F'|' 'NF == 10 && $9 == "87001"' /var/spool/xi/batch/varaudit.log

Deletions, whose value field is empty:

awk -F'|' '$4 ~ /(^|:)delete$/' /var/spool/xi/batch/varaudit.log

Logging to Programs

A LOGVARS value beginning with | is a shell command line. The scheduler starts it through the shell and writes each log line to its standard input. Four properties govern such a sink, and they are the same as for LOGJOBS, where they are described in full: only a local administrator may set one, a network peer or a job being refused; it runs with the scheduler's privileges, so give an absolute path to a script other users cannot write; it is closed and restarted every time the scheduler rewrites its queue files, at most every 300 seconds, so repeated invocation must be harmless; and if it stops reading, entries are discarded with no message. Make no network calls from a sink - the scheduler blocks on it.

A variable-log sink must also cope with the field count changing from line to line.

Example state monitor:

#!/bin/sh
# /usr/local/bin/var-logger.sh
# Runs from the scheduler. Keep it fast; it is re-run periodically.

while IFS='|' read -r date time varname status event user group value jobnum jobtitle; do
    # Strip any machine: prefix from the status field
    code=${status##*:}

    # Alert on a critical state change
    if [ "$varname" = "SYSTEM_STATUS" ] && [ "$code" = "assign" ] && [ "$value" = "ERROR" ]; then
        logger -t xibatch "SYSTEM_STATUS set to ERROR by $user (${event}${jobnum:+, job $jobnum})"
    fi

    # Append every entry to a file of our own
    printf '%s|%s|%s|%s|%s|%s|%s|%s|%s|%s\n' \
        "$date" "$time" "$varname" "$status" "$event" \
        "$user" "$group" "$value" "$jobnum" "$jobtitle" \
        >> /var/log/xibatch-vars.log
done

On an eight-field line, jobnum and jobtitle are left empty by read, which is what the test above relies on.

Use Cases

Workflow debugging

Trace variable state changes through complex job dependencies

State machine monitoring

Track workflow state transitions for analysis

Audit compliance

Record who changed critical control variables

Race condition detection

Identify conflicting variable updates from concurrent jobs

Performance analysis

Monitor variable usage patterns to optimise conditions

Integration verification

Confirm external systems are setting variables correctly

Correlating Job and Variable Logs

The job number is the only field the two logs share, and it sits in a different place in each: field 3 of the job log, field 9 of the variable log. Both are bounded by separators, so one pattern finds a job in either.

Example correlation script:

#!/bin/sh
# Show job and related variable changes

JOBNUM=$1

echo "=== Job Activity ==="
grep "|$JOBNUM|" /var/spool/xi/batch/jobaudit.log

echo ""
echo "=== Variable Changes ==="
grep "|$JOBNUM|" /var/spool/xi/batch/varaudit.log

Output shows:

=== Job Activity ===
06/02/2026|10:15:22|87001|Process data|started|jmc|users|150|1000
06/02/2026|10:18:45|87001|Process data|completed|jmc|users|150|1000

=== Variable Changes ===
06/02/2026|10:15:22|cnt|assign|Job start|jmc|users|2012|87001|Process data
06/02/2026|10:18:45|STATUS|assign|Job completed|jmc|users|Done|87001|Process data

The job log's status codes are lower case and its field order is different; see the LOGJOBS article for that format. A job with no title appears as <Unnamed job> in the job log and as an empty field in the variable log, which is worth allowing for when joining the two.

Monitoring Critical Variables

Alerting on a particular variable is a matter of filtering the log. Do the alerting from a process that reads the log file rather than from a LOGVARS sink, for the reasons given above.

#!/bin/sh
# Report LOADLEVEL changes from the variable log

awk -F'|' '$3 == "LOADLEVEL" && $4 ~ /(^|:)assign$/ {
        printf "%s %s  LOADLEVEL = %s  (%s, by %s)\n", $1, $2, $8, $5, $6
    }' /var/spool/xi/batch/varaudit.log

The current load, CLOAD, does not appear in this log at all - the scheduler maintains it directly. Read it with btvar CLOAD instead.

Log Rotation

Variable logs grow without limit, and Xi-Batch never rotates the log and never reopens it. The scheduler holds the file open until the value changes or it stops, and it ignores SIGHUP, so the usual rename-and-create recipe leaves the scheduler appending to the renamed file while the new one stays empty, with nothing reporting the loss. Do not give logrotate a create stanza for this file.

Two arrangements work, and are described in full in the LOGJOBS article: rotate with copytruncate, which is safe because the file is opened in append mode; or move the file and re-assign the variable, which closes the old file and opens a new one with the right owner and mode:

#!/bin/sh
# Run as root or as the batch user
LOG=/var/spool/xi/batch/varaudit.log
DATE=$(date +%Y%m%d)

if [ -f "$LOG" ]; then
    mv "$LOG" "${LOG}.${DATE}"
    btvar -s varaudit.log LOGVARS
    gzip "${LOG}.${DATE}"
fi

Analysing Variable Usage Patterns

Identify frequently-changed variables:

#!/bin/sh
# Variable change frequency report

echo "Variable Change Frequency"
echo "========================="

awk -F'|' '{ n[$3]++ }
    END { for (v in n) printf "%-20s %d\n", v, n[v] }' \
    /var/spool/xi/batch/varaudit.log | sort -k2 -rn

The date field is day-first, so it cannot be compared as a string to select a period - 01/02/2026 sorts before 28/01/2026. To report on a fixed period, rotate the log on that period and run the report against one rotated file, or convert the field to yyyymmdd before comparing.

Network-Wide Variable Tracking

In a networked installation each scheduler logs only the changes it applies itself. A change to an exported variable is logged on the machine that owns the variable; the copies held by other machines are updated by broadcast and produce no entries there. To follow an exported variable across a network, read the log on its owning host.

Find changes requested from another host:

awk -F'|' '$4 ~ /:/ {print}' /var/spool/xi/batch/varaudit.log

The machine: prefix on field 4 marks a request that arrived over the network, or an assignment made by a job that ran on another host.

List the exported variables to watch for:

btvlist -N -F '%N|%E'

The %E column is non-empty for an exported variable. Use %K in place of %E for cluster variables.

Verifying Logging is Active

Check the LOGVARS value:

btvar LOGVARS

btvar with no operation options prints the variable's current value; empty output means logging is off.

Test logging:

# Create a test variable with value 100
btvar -C -s 100 test_var

# Change it
btvar -s 200 test_var

# Delete it
btvar -D test_var

# Check the log
tail /var/spool/xi/batch/varaudit.log

You should see create, assign and delete entries for test_var. Note that -C creates and -D deletes; the lower-case -c sets the comment.

Troubleshooting

No variable changes logged

Check the value with btvar; an empty value means logging is off, and a fresh variable file or a migration is the usual reason. If the value is set, confirm the batch user can create and write the file in the spool directory. A destination the scheduler cannot open produces no error message anywhere, including in btsched_reps - logging is simply inactive.

A change you know happened is missing

it was probably applied on another machine. Changes broadcast from the host that owns an exported variable, and requests you make against machine:variable, are logged on that host rather than this one.

Incomplete log entries

entries are eight fields for a change made outside a job and ten for a change made by one, and the value field is empty on a delete. Check that the parser is not assuming a fixed field count before assuming a truncated line.

Missing job information in logs

the job number and title are present only for job-driven changes, and the title is empty when the job has no title.

Log file permissions errors

the file's owner, group and mode come from the LOGVARS variable. Grant read on the variable with btvar -M, then re-assign the value so the change reaches the file.

Entries stopped after a rotation

the scheduler is still writing to the rotated file. Re-assign LOGVARS to reopen the current path.

Combining LOGJOBS and LOGVARS

Enable both for a complete audit trail:

# Enable job logging
btvar -s jobaudit.log LOGJOBS

# Enable variable logging
btvar -s varaudit.log LOGVARS

# Check both
btvar LOGJOBS
btvar LOGVARS

btvar takes one variable name per invocation. Both are set as separate assignments, and each takes effect as it is applied.

Best Practices

Set appropriate permissions

Ensure LOGJOBS and LOGVARS variables have restrictive modes to protect log files

Implement rotation

Prevent disk space exhaustion with copytruncate or the move-and-re-assign method, never a plain create

Monitor log programs

Ensure logging programs don't block scheduler operations

Correlate logs

Use job numbers to link job events with variable changes

Archive logs

Retain logs for compliance requirements (90 days, 1 year, etc.)

Filter important events

When logging to programs, focus on significant changes to reduce noise

Test logging

Verify logging works after enabling by making a test change and checking the log

Document log format

Ensure the operations team knows the field count varies between eight and ten