Integrating Xi-Batch With Other Systems

What Xi-Batch Actually Offers an External System

Xi-Batch has five integration surfaces. Everything else an external system does with it is built out of these.

There is no HTTP interface, no REST API, no JSON, no webhook receiver, no message-queue connector and no plugin mechanism. There is no scheduler-side hook that runs a program of your choice when a job finishes. Anything of that shape has to be built out of the surfaces above, in a program you write and schedule yourself.

Two facilities that look like exceptions are covered at the end: a file-arrival monitor that ships only in the tarball, and a CGI web interface that ships nowhere.

Driving Xi-Batch From the Command Line

This is the surface that is present on every installation, and for most integrations it is the whole answer.

Submitting a Job

btr -h 'Order batch 8821' -T 20:15 /apps/process-order.sh

Three things about btr trip up code that calls it:

• The run time is -T and the title is -h. -t is the delete time and is accepted silently, so a job given -t instead of -T runs at once and then disappears. There is no relative or English time syntax such as "now + 5 minutes" or "tomorrow": the argument must begin with a digit. A bare hh:mm means today; the full form is yy/mm/dd,hh:mm, with a comma between the date and the time rather than a space. A two-part date is read as day/month or month/day depending on the machine's timezone, so scripts should give all three parts.
• btr copies the file's contents into the queue. The job is a snapshot taken at submission; editing the file afterwards does not change the job. With no file argument it reads standard input, which is how a generated script is submitted without writing it to disk.
• The script's #! line is a comment. The job text is handed to the job's command interpreter on standard input, so a #!/bin/bash header does nothing. If the job needs a particular interpreter, name it: btr -i ksh, and add the interpreter to the list first if it is not there.

A repeating job takes -r with one of Minutes, Hours, Days, Weeks, Monthsb, Monthse or Years and a rate:

btr -h 'Poll for work' -T 08:00 -r Minutes:5 /apps/poll-queue.sh

Asking Whether a Job Has Finished

The command is btjstat. It prints nothing and answers through its exit status, which is what makes it usable from a script:

btjstat -s Done 1234 && run-next-step.sh

btjstat [-d] [-s states] <job number>

-s takes a comma-separated list of state names and matching is not case-sensitive. The states are Done, Err, Abrt, Canc, Init, Strt, Run and Fin; a job that has never run displays as an empty state and matches none of them. With no -s the test is whether the job is in progress - Init, Strt, Run or Fin - so a completed job answers "no".

The exit statuses are 0 for a match, 1 for no match, 2 for bad arguments, 6 if the scheduler is not running and 13 if there is no such job. A caller that treats any non-zero status as "not finished" will loop for ever on a job that has been deleted, so distinguish 1 from 13.

For anything richer, use btjlist with an explicit format rather than parsing the default columns:

btjlist -N -F '%N|%H|%P|%x'

-N suppresses the heading row, %N is the job number, %H the title, %P the state and %x the exit code of the last run. Choosing your own separator is worth doing: titles contain spaces.

Reading and Setting Variables

Reading takes no option. btvar with a bare name prints the value on one line and nothing else:

state=`btvar integration_status`

Writing puts the value on -s and leaves the name positional - the opposite way round from what most people write first:

btvar -s 'Ready' integration_status
btvar -s 0 error_count

Both need a running scheduler and both are subject to the variable's permissions.

To see the whole set with their export state, use a format string; the default listing does not show the cluster column:

btvlist -N -F '%N|%V|%E|%K'

Job Outcome: the Exit Code Is the Interface

Every job's exit code is turned into one of three recorded states, and this is the most reliable signal an external system can read.

Both ranges are per job and can be changed:

btr -X N0:2 -X E3:9 -h 'Tolerant job' /apps/extract.sh

Here 0, 1 and 2 all count as success, 3 to 9 as an error, and anything else as an abort. -X takes N or E followed by a single number or a low:high pair, and the numbers must be 0 to 255.

This matters beyond reporting: a repeating job that ends outside its normal range keeps its error state permanently and stops repeating, with no message. An integration that submits repeating work should widen the normal range to cover the codes it expects, or check the state on a schedule.

Two special assignment values carry the outcome into a variable, so an external system can read it without knowing the job number:

btr -h 'Nightly extract' -T 01:00 -s 'extract_rc=exitcode' /apps/extract.sh
btr -h 'Nightly extract' -T 01:00 -s 'extract_sig=signal'  /apps/extract.sh

Both fire at completion, error and abort whatever stage flags are set.

Variables as the State Channel

A variable is the product's own mechanism for one job to wait on another, and for an external system to take part in that.

An External System Signalling Xi-Batch

Create the variable once, then have the external system set it and have the job wait on it:

btvar -C -s 'No' -c 'Set by the order feed when today file is complete' feed_ready

btr -h 'Process order feed' -c 'feed_ready=Yes' /apps/process-feed.sh

A condition takes no spaces. The name, the operator and the value are one unbroken string: feed_ready=Yes parses, feed_ready = Yes is rejected with Bad condition. The operators are =, !=, <, <=, > and >=.

A job held by an unmet condition waits indefinitely and nothing is recorded when it is skipped, so an integration that depends on a condition should also have a way of noticing that the job has not run.

Xi-Batch Signalling an External System

The -s assignment list is how a job reports progress into a variable that something else polls. Each -f chooses the stages for the -s options that follow it:

btr -h 'Extract'   -T 01:00 -f S -s 'pipeline=extracting'  -f N -s 'pipeline=extracted'  /apps/extract.sh
btr -h 'Transform' -c 'pipeline=extracted' -f S -s 'pipeline=transforming' -f N -s 'pipeline=transformed' /apps/transform.sh
btr -h 'Load'      -c 'pipeline=transformed' -f S -s 'pipeline=loading' -f N -s 'pipeline=loaded' /apps/load.sh

The stage letters are S at start, N on normal completion, E on error, A on abort, C on cancellation and R to reverse the assignment when the job ends.

The default is the trap here. With no -f, an assignment is given S, R, N, E and A - it fires at start and is undone at completion, error and abort. A counter written that way returns to its old value the moment the job finishes, and a watcher polling once a minute sees nothing. Set -f explicitly on anything an external system reads.

Let the scheduler do arithmetic rather than reading a value and writing it back: -s 'runs+=1' is applied in one operation, where a read-modify-write from a script can lose an update.

The full set of operators is =, +=, -=, *=, /= and %=, and the same no-spaces rule applies.

Being Told That a Job Finished

Xi-Batch will notify the job's owner, and only the owner:

btr -m -h 'Nightly extract' -T 01:00 /apps/extract.sh   # mail on completion
btr -w -h 'Nightly extract' -T 01:00 /apps/extract.sh   # write to the owner's terminal

Three details decide whether this is usable as an integration point:

• Mail is sent anyway if the job produced output. If neither flag is set and the job wrote to standard output or standard error without redirecting them, the notification is forced to mail so that the output is not lost.
• The terminal message is suppressed while the owner is in btq, on the reasoning that they can already see the queue.
• A job whose owner is not in the local password file gets no notification at all. Such owners are displayed as a u followed by digits, and the notification is abandoned silently.

There is no way to direct the notification anywhere else - no address setting, no command hook. If a monitoring system is to be told, the job's own script has to tell it, or a separate polling job has to.

Submitting Into Another Host's Queue

btr -Q prod2 -h 'Remote extract' -T 01:00 /apps/extract.sh

This does not use the scheduler-to-scheduler connection. It re-runs the request through rbtr, which talks to the external server on the target host over the xbnetsrv service, so it needs that host to hold a network licence and that port to be reachable. It works whether or not the two schedulers are connected to each other.

The complementary route - one job that any connected host may run - is the remote runnable export state, set with btr -G or btjchange -G. Which host picks it up depends on which has spare load capacity and has the job's command interpreter defined; there is no dispatcher and no way to choose. See the networking article for the setup and the load-level article for the capacity rule.

The C API

Xi-Batch has a real programming interface: a library, libxbapi, and a header, xbapi.h, with its own reference manual. A program links against it and gets direct access to jobs, variables, command interpreters, the user permissions and the holiday calendar, with functions such as xb_jobadd, xb_joblist, xb_jobupd, xb_jobdel, xb_varadd, xb_varread, xb_varupd, and monitor calls that block until a job or a variable changes.

A session is opened with one of:

xb_open(hostname, servname)
xb_login(hostname, servname, username, password)
xb_locallogin(servname, username)

and the default service name is xbapi, which is the entry the installation adds to /etc/services. The server side is the same daemon that serves the Windows clients; it is started by btstart and only when the licence carries the network flag. So the API needs a network licence even for a program running on the same machine.

The library and the header are installed only by the tarball distribution. Neither the RPM nor the Debian package contains them - the package ships the server side and not the client side. On a packaged installation there is no supported way to compile against the API, and the practical answer is the command line. Raise it with support if you need the API on a packaged host.

Watching for a File to Arrive

The usual answer to "run something when a file lands" is a polling loop. Xi-Batch has a program for it, btfilemon, which watches a directory and runs a script or a command when a file matching a pattern arrives, is deleted, or stops growing, stops being written to, stops changing or stops being used for a given number of seconds:

btfilemon -d -D /data/incoming -p '*.trigger' -A -R -X /apps/on-trigger.sh

It runs detached with -d, applies recursively with -R, ignores files already present with -i, and can list and kill its own monitor processes with -l, -k and -K.

It is installed only by the tarball distribution, like the API library. On an RPM or Debian installation you have to poll instead, from a job:

btr -h 'Trigger watcher' -T 08:00 -r Minutes:1 /usr/local/sbin/file-watcher.sh

Write the watcher to move the trigger file aside before acting on it, so that a run that overlaps its successor cannot process the same file twice.

What Does Not Exist

Stated plainly, because each of these is a reasonable thing to look for and finding nothing is otherwise indistinguishable from looking in the wrong place.

• No HTTP or REST interface, and no JSON anywhere in the product. Nothing in Xi-Batch listens for or emits an HTTP request.
• No webhooks in either direction. A job can of course run curl, because a job can run anything; that is your script calling out, and the scheduler will retry it only by re-running the whole job.
• No message-queue connector and no plugin mechanism. The command interpreter list decides which program runs a job's text; it is not an extension point for the scheduler itself.
• No completion hook. The scheduler runs no program of your choosing when a job ends. What it does is set the job's state, apply the assignment list and, if asked, notify the owner.
• No log-shipping facility. The scheduler's report file is written by appending to a descriptor it opens once and never reopens, and the product ignores SIGHUP, so the usual rename-and-create rotation leaves the scheduler writing to the renamed file. Copy and truncate instead, and ship the copy.
• The CGI web interface is not something you can deploy. A set of CGI programs and an old manual for them exist, but no installation route - tarball, RPM or Debian package - installs them, and the HTML templates they render are not in the distribution at all. Treat any reference to a browser interface as historical.

Writing Jobs That Other Systems Drive

Do not rely on the shebang line. Covered above: the interpreter comes from the job, not from the script. Choose it with btr -i.

Keep the script portable if the site is not Linux-only. Xi-Batch runs on Solaris, AIX and HP-UX as well as Linux, and a good deal of integration boilerplate does not: ps aux is BSD syntax and needs ps -ef; date -Iseconds, logger -n and nc are GNU or Linux-specific; bash is often absent. The scheduler's own generated scripts are strict /bin/sh for the same reason.

Give every job a title. It is what an external system can match on, since job numbers are allocated fresh and are not preserved by a backup and restore.

Redirect standard output and standard error deliberately. Left alone they accumulate in the spool directory and force a mail to the owner on every run.

Write the caller's identity into a variable or into the job's title. Nothing in the job record says where a submission came from, and a comment in the script text is not something any listing shows.

Practices

Prefer a variable to a file for state that crosses the boundary. A variable has an owner, permissions, a comment, an audit trail through the variable log, and arithmetic the scheduler applies atomically.

Set -f explicitly on any assignment something else reads. The default reverses itself when the job ends.

Poll with btjstat, not by parsing btjlist. It is a single exit status and it will not change shape.

Give any waiting loop a limit. Nothing in Xi-Batch times out a condition, a held job or an external system that never answers.

Widen the normal exit range rather than letting a repeating job stop. A repeating job that records an error stays in that state until someone clears it.

Keep integration scripts in version control, and re-submit the job after editing one. The queue holds a copy taken at submission; the file on disk and the job can silently diverge.