Tracking Job Activity with LOGJOBS

Understanding Job Audit Trails

The LOGJOBS system variable controls job activity logging in Xi-Batch. When enabled, Xi-Batch records every significant job event to a file or program, creating a comprehensive audit trail of scheduling activity.

What Gets Logged

Job logging captures these lifecycle events:

Job creation : When jobs are submitted to the queue

Job starts : When scheduler begins executing a job

Job completion : When jobs finish successfully

Job errors : When jobs exit with error codes

Job abort/cancel : When jobs are forcibly terminated

Job modifications : When job parameters change (owner, group, mode, details)

Force operations : When jobs are forced to run immediately

Enabling Job Logging

Set the LOGJOBS variable to specify log destination:

Log to file:

bash

btvar -s LOGJOBS jobaudit.log

File path is relative to spool directory (default /var/spool/xi/batch). Full path jobaudit.log becomes /var/spool/xi/batch/jobaudit.log.

Log to program/script:

bash

btvar -s 'LOGJOBS |/usr/local/bin/job-logger.sh'

Program name must start with | character. The PATH from scheduler startup is used to find the program.

Disable logging:

bash

btvar -s 'LOGJOBS '

Empty string turns off logging.

Log File Permissions

When logging to a file, permissions match the LOGJOBS variable settings:

File owner : Same as LOGJOBS variable owner

File group : Same as LOGJOBS variable group

File modes : Read/write permissions from LOGJOBS variable

This ensures log files have appropriate access control matching the variable that controls them.

Log Entry Format

Each log entry is a single line with pipe-delimited fields:

05/01/99|10:22:43|13741|date|completed|jmc|users|150|1000

Field order:

1. Date - dd/mm/yy or mm/dd/yy (depends on timezone)
2. Time - HH:MM:SS
3. Job Number - Local job number or machine:jobnumber for remote jobs
4. Job Title - Job name or <unnamed job> if no title
5. Status Code - Event type (see below)
6. User - Job owner username
7. Group - Job group name
8. Priority - Job priority value
9. Load Level - Job load level value

Status codes prefixed with machine: for events from remote hosts.

Status Codes

Abort : Job was aborted (forcibly terminated while running)

Cancel : Job was cancelled (removed before or during execution)

Chgrp : Job group was changed

Chmod : Job permissions (modes) were changed

Chown : Job owner was changed

Completed : Job finished successfully (exit code 0)

Create : Job submitted to queue

Delete : Job removed from queue

Error : Job finished with error (non-zero exit code)

force-run : Job forced to run without advancing scheduled time

force-start : Job forced to start immediately

Jdetails : Job details modified (conditions, assignments, time, etc.)

Started : Job execution began

Processing Log Files

The pipe-delimited format facilitates easy parsing:

Count jobs by status:

bash

awk -F'|' '{print $5}' /var/spool/xi/batch/jobaudit.log | sort | uniq -c

Find failed jobs:

bash

grep '|Error|' /var/spool/xi/batch/jobaudit.log

Track specific user's jobs:

bash

awk -F'|' '$6 == "jsmith"' /var/spool/xi/batch/jobaudit.log

Daily job completion count:

bash

awk -F'|' '$5 == "Completed" {print $1}' jobaudit.log | sort | uniq -c

Jobs with high load levels:

bash

awk -F'|' '$9 > 5000' /var/spool/xi/batch/jobaudit.log

Logging to Programs

When LOGJOBS specifies a program (starting with |), each log line is sent to the program's standard input:

Example logger script:

bash

#!/bin/bash
# /usr/local/bin/job-logger.sh

# Read log entries from stdin
while IFS='|' read -r date time jobnum title status user group priority load; do
    # Send high-priority errors to syslog
    if [[ "$status" == "Error" && "$priority" -gt 150 ]]; then
        logger -t xibatch "High priority job failed: $title (job $jobnum, user $user)"
    fi
    
    # Write all entries to file
    echo "$date|$time|$jobnum|$title|$status|$user|$group|$priority|$load" \
        >> /var/log/xibatch-jobs.log
done

Important: Ensure logging programs don't block. The scheduler waits for the program to process each line. Slow logging programs delay job operations.

Use Cases

Compliance auditing : Track who ran what jobs when for regulatory requirements

Performance monitoring : Identify frequently failing jobs or resource-intensive jobs

Capacity planning : Analyse job patterns to optimise load levels and scheduling

Troubleshooting : Review job history when investigating scheduling issues

Billing/chargebacks : Track resource usage by user or group

Security monitoring : Detect unusual job submission patterns

Log Rotation

Job logs grow continuously. Implement rotation to manage size:

Using logrotate (Linux):

bash

# /etc/logrotate.d/xibatch-jobs
/var/spool/xi/batch/jobaudit.log {
    weekly
    rotate 52
    compress
    delaycompress
    missingok
    notifempty
    create 0640 batch batch
}

Manual rotation script:

bash

#!/bin/bash
LOG="/var/spool/xi/batch/jobaudit.log"
DATE=$(date +%Y%m%d)

if [ -f "$LOG" ]; then
    mv "$LOG" "${LOG}.${DATE}"
    gzip "${LOG}.${DATE}"
    touch "$LOG"
    chown batch:batch "$LOG"
    chmod 640 "$LOG"
fi

Integration with Monitoring Systems

Send job events to external systems:

Example integration script:

bash

#!/bin/bash
# Send job failures to monitoring system

while IFS='|' read -r date time jobnum title status user group priority load; do
    if [[ "$status" == "Error" ]]; then
        # Send to monitoring API
        curl -X POST https://monitoring.example.com/api/events \
            -H "Content-Type: application/json" \
            -d "{
                \"type\": \"job_failure\",
                \"job\": \"$title\",
                \"user\": \"$user\",
                \"timestamp\": \"$date $time\"
            }"
    fi
done

Verifying Logging is Active

Check LOGJOBS value:

bash

btvar -v LOGJOBS

Test logging:

bash

# Submit test job
echo "echo test" | btr -t "Test logging"

# Check log file
tail /var/spool/xi/batch/jobaudit.log

Should see entries for job creation, start, and completion.

Troubleshooting

No log entries appearing : Verify LOGJOBS is set to non-empty value. Check file permissions. Ensure spool directory writable.

Log program not receiving data : Verify program path is correct. Check program is executable. Review scheduler log (btsched_reps) for errors.

Log file grows too large : Implement log rotation. Consider logging to program that filters important events only.

Permission denied errors : LOGJOBS variable permissions control log file access. Ensure variable has appropriate modes.