Running Xi-Batch on Solaris, AIX and HP-UX

Understanding Platform Differences

Xi-Batch runs across several Unix and Linux platforms. The commands, options and file layout are the same everywhere; what differs is how the operating system starts services at boot, how it accounts for users, and - the one that matters most for a job scheduler - where it keeps the shells and interpreters that jobs are run by. This article is about running Xi-Batch on a platform other than the one you are used to, and about the settings that have to be revisited when a scheduler moves from one platform to another.

One rule underlies all of it: the product is installed as a binary built for one platform, and a configuration moves between platforms as text produced by the conversion tools, never as a copy of the spool directory's state files.

Supported Platforms

These are the platforms a system may be registered against on the customer portal, and therefore the platforms for which a build can be issued:

Linux

Debian 12 and 13, Rocky Linux 8 and 9 (and other Red Hat rebuilds), SUSE Linux Enterprise 15 / openSUSE Leap, Ubuntu 22.04 and 24.04, on x86_64; Debian 13 and Rocky Linux 9 also on aarch64

Solaris

Solaris 10 and 11 on SPARC, Solaris 11 on x86_64

AIX

AIX 7.1, 7.2 and 7.3 on POWER

HP-UX

HP-UX 11.23 and 11.31 on Itanium, HP-UX 11.23 on PA-RISC

Older releases of these operating systems, and platforms retired years ago, are served from the archive of historical builds rather than from current packages. If a platform is not offered when you register a system on the portal, there is no current build for it - ask Xi Software before planning a migration onto it.

Obtaining the Right Build

Builds are downloaded from the customer portal. Register the machine as a system, choosing its architecture, operating system and version; the portal then lists the builds that match it. Once a system is registered its platform fields are fixed, so a machine that changes platform is registered afresh rather than edited.

On Linux the product is supplied as an RPM or a Debian package, according to the distribution family. On Solaris, AIX and HP-UX it is supplied as a tar.gz containing an interactive installer.

Directory Conventions

Base Directories

The compiled-in defaults are the same on every platform:

• Spool directory: /var/spool/xi/batch
• Internal programs: /usr/libexec/xi
• User commands: /usr/local/bin

Installations that use /usr/spool are older ones that were placed there when /usr/spool was the convention, on Linux as much as on the traditional Unix platforms. The location is a property of the installation, not of the platform. Read the installation's own master configuration file, /etc/xi/batchconfig, rather than assuming either layout.

Choosing Different Locations

The packaged installations use the defaults above. The tar.gz installer asks whether you want to change the location of the main directories, and then whether to base them all on a common directory. There is no command-line option for this - the installer takes no arguments and asks its questions interactively.

User Binary Locations

The user commands are installed in /usr/local/bin. Ensure that directory is on the PATH of everyone who uses the product, and of any startup script that calls btstart or btquit - several platforms omit it from root's default PATH.

User and Group Management

User Creation

You do not normally create the system user by hand. The packaged installations create it in their pre-install step, and the tar.gz installer prints "I need to install user ..." and asks to continue, then creates it with the first free user id from 51. The Xi-Batch user is batch.

Because the id is chosen from whatever is free on each host, the same account will usually have different numeric ids on different machines. That is expected and does not need correcting.

If you must create the account yourself before installing - on a host where accounts come from a directory service, for example - create it with the name the product expects and a home directory under the spool area, then run the installation.

Password and Group Databases

Xi-Batch builds its own table of user names and ids by reading /etc/passwd and /etc/group directly, line by line, rather than through the system's name-service library. This has a consequence that catches sites out when they move onto a platform with centralised accounts:

NIS, LDAP and other directory services are not consulted. A user who exists only in a directory service is not found in that table, and is displayed and exported by number instead of by name, as u1234. Nothing fails; the name is simply wrong everywhere it appears, including in the scripts the conversion tools generate.

If you need names to appear, the accounts that use the product must be present in the local /etc/passwd on the machine running it. This is a good reason to keep the Xi accounts local even where the rest of the estate is centralised.

Shell and Command Interpreters

This is the platform question with the most work in it for Xi-Batch, because a job is run by an interpreter and the interpreters live at different paths on different platforms.

Default Shells by Platform

Linux

/bin/bash typically default

Solaris

/bin/sh (POSIX shell) or /bin/bash

HP-UX

/bin/sh (POSIX), /bin/ksh available

AIX

/bin/ksh (Korn shell) typically default

Which Shell the Product Uses

Xi-Batch uses a standard shell of its own, which is fixed when the build is made - it is the sh found on the machine the build was produced on. You can override it for the installation by setting SHELL in the master configuration file, /etc/xi/batchconfig. That is the setting to check first if jobs fail with a "not found" error immediately after a platform change.

The Command Interpreter Table

An Xi-Batch job names a command interpreter, and the interpreter's path is held in the interpreter table, not in the job. When the platform changes and an interpreter has moved - /bin/ksh on one system, /usr/bin/ksh on another - correct the table with btcichange and every job that names that interpreter is corrected with it. List the current table with btcilist:

btcilist

Editing a "#!" line inside a job script is the wrong lever: the job is handed to the interpreter the table names for it, and the line inside the script has no say in that.

Restore or correct the interpreter table before starting the scheduler. btcichange edits the interpreter file directly and a running scheduler picks up a change without a restart, so the requirement is not about the scheduler noticing. It is about the job file: the scheduler loads the saved jobs straight after the interpreter list, and any saved job naming an interpreter that is missing at that moment is silently reset to the first entry in the table. Once that has happened, correcting the table afterwards does not put the jobs back.

Generated Scripts

The scripts produced by the conversion tools begin "#! /bin/sh" and use POSIX constructs only, so they replay on every supported platform. Run them with sh; bash is not present by default on stock AIX or HP-UX.

IPC Facilities

What the Product Actually Uses

A running Xi-Batch uses:

• System V shared memory - for the job queue and the variable list. A further segment appears briefly while a segment is being grown.
• File locks - advisory record locks taken with fcntl() on lock files in the spool directory. No System V semaphores are used, except one set when a networked licence is in force.
• A Unix-domain socket for requests to the scheduler. No System V message queue is used.

All four platforms provide these facilities, so nothing here differs by platform in kind. What differs is the limit each kernel places on shared memory.

Shared Memory Sizing

The size of the segments is decided by the job and variable counts given when the product is started, as in btstart -j 2000 -v 500. Asking for a large queue on a system with a low shared-memory limit is what causes a failure to start, and it is reported as a message naming the segment and the system error.

If you see that, either reduce the requested counts or raise the kernel's shared-memory limit using the platform's own tuning mechanism. The default limits on current Linux kernels are large enough for any realistic queue; the older Unix platforms are the ones where tuning is more often needed. Consult the platform's documentation for the current mechanism, as these have changed between releases of every one of these operating systems.

Networking

Service Files

All platforms use /etc/services for service-to-port mapping. The installation adds its own entries if they are not already present. Xi-Batch adds:

xibatch         2050/tcp        # Connection port
xibatch         2050/udp        # Probe port
btq             2150/tcp        # Feeder port
xbnetsrv        2250/tcp        # External job submission
xbnetsrv        2250/udp        # Client access
xbapi           2260/tcp        # Xi-Batch API
xbapi           2260/udp        # Xi-Batch API

A site may change these, and the product reads the port from /etc/services rather than assuming it, so every machine in a networked installation must agree. Copy the entries rather than retyping them, and open the firewall for the ones you use.

Host Resolution

The network configuration lives in /etc/xi/batch-hosts. It names the other machines in the installation, and each name is resolved through the system resolver, so /etc/hosts, DNS or a mixture all work provided the answer is consistent on every machine.

Two constraints catch people out when this file is moved between platforms:

• Host addresses are resolved as IPv4. A name that resolves only to an IPv6 address will not be usable.
• The file may begin with a localaddress line giving the machine's own address, and if present it must be the first non-comment line in the file. A localaddress line further down is ignored and the file is flagged as having errors.

Because of the localaddress rule, this file is not simply copied identically to every machine. Keep one master list of the other hosts and prepend each machine's own localaddress line.

Terminal Handling

The character-mode screen, btq, is a curses program and uses whichever terminal database the platform's curses library was built against, terminfo on current systems and termcap on older ones. Ensure a definition exists for the terminal types in use, and set the TERM environment variable appropriately - TERM=vt100 or TERM=wyse60 for a physical terminal, TERM=xterm or TERM=screen for an emulator, and whatever the client sends over SSH.

A TERM value the target platform does not have a definition for causes display corruption in btq. Terminal databases differ between platforms, so a TERM setting that works on Linux may have no entry on AIX or HP-UX.

File System Differences

Maximum Path Length

The product falls back to a limit of 1024 characters where the platform does not define one. Spool file names and job work directories are all built within that limit, so a deeply nested base directory reduces the room left for the rest.

Maximum Job Size

Job sizes and file positions are held as 32-bit quantities in the saved formats, so a single job is limited to just under 2 GB. This is a property of the format and applies equally on a 64-bit build and a 64-bit platform.

File Locking

Locking uses fcntl() advisory record locks, which every supported platform provides. BSD-style flock() is not used, so its absence on a platform is not a concern.

NFS considerations

Advisory record locking over NFS depends on a working lock manager on both sides and behaves inconsistently in practice. Keep spool directories on local file systems.

File System Types

Local file systems (ext4, xfs, zfs, jfs2, vxfs)

Suitable for spool directories

Network file systems (NFS, SMB/CIFS)

Unsuitable, because of the locking behaviour above

Temporary file systems (tmpfs, ramfs)

Unsuitable - the queue would be lost at every reboot

Character Encoding

The scheduler and its command-line tools do not set a locale, so they run in the C locale whatever LANG or LC_ALL are set to. A job's own environment is a separate matter: what a job's script does with LANG is between the script and the programs it runs, and the scheduler takes no part in it.

System Startup Integration

What the Installation Sets Up

The RPM and Debian packages install a systemd unit and enable it. The unit runs the product's own start and stop commands:

ExecStart=/usr/local/bin/btstart -j 2000 -v 500 -l 10000
ExecStop=/usr/local/bin/btquit -y

Adjust the numbers on the ExecStart line to size the queue for the site, then reload systemd.

The tar.gz installer detects the platform's startup mechanism and writes the appropriate file:

systemd (current Linux)

a unit file, which the installer also enables

Upstart (older Ubuntu)

a job file under /etc/init

LSB and System V init (older Linux, Solaris)

/etc/init.d/xibatch plus rc links, registered with update-rc.d or chkconfig where available

AIX

a line appended to /etc/inittab, naming the start command and the run level

HP-UX

the installer prints a message that it does not know how to handle HP-UX startups and installs nothing. Add the startup by hand as below.

Whatever the platform, verify after installation that the scheduler actually starts on boot - reboot a test machine rather than assuming.

Adding Startup by Hand

On a System V style system:

# Create startup script
cat > /etc/init.d/xibatch << 'EOF'
#!/bin/sh
case "$1" in
start)
    /usr/local/bin/btstart -j 2000 -v 500 -l 10000
    ;;
stop)
    /usr/local/bin/btquit -y
    ;;
esac
EOF

chmod 755 /etc/init.d/xibatch
ln -s /etc/init.d/xibatch /etc/rc3.d/S80xibatch
ln -s /etc/init.d/xibatch /etc/rc0.d/K01xibatch

Use the run level the system actually boots to - the initdefault entry in /etc/inittab - rather than assuming 3, and start the product late (S80 above) so that the network is already up.

Those paths are the Solaris and older-Linux ones. On HP-UX the startup directories are under /sbin, not /etc: write the script as /sbin/init.d/xibatch and link it from /sbin/rc3.d and /sbin/rc0.d. HP-UX is the platform the installer leaves alone, so this is the case that most often has to be done by hand.

On a systemd system, if you are adding a unit rather than taking the packaged one:

# Create unit file
cat > /etc/systemd/system/xibatch.service << 'EOF'
[Unit]
Description=Xi-Batch Job Scheduler
After=network.target

[Service]
Type=forking
ExecStart=/usr/local/bin/btstart -j 2000 -v 500 -l 10000
ExecStop=/usr/local/bin/btquit -y
Restart=on-failure
TimeoutStopSec=330
KillMode=mixed

[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload
systemctl enable xibatch
systemctl start xibatch

Type=forking is required - btstart returns once the daemon is running. The generous TimeoutStopSec matters: a graceful stop syncs the spool before releasing the shared memory, which can take longer than systemd's default, and a scheduler killed part way through leaves its shared memory segments behind.

What a Platform Change Costs You

Two things do not survive a move to different hardware, whatever the platform, and both are properties of the machine rather than of the configuration.

The licence. It is keyed to the machine's hardware address and to the word size of the build, and it records neither the operating system nor the host name. Renaming a machine leaves a working licence; moving to different hardware does not, and replacing a 32-bit installation with a 64-bit one needs new codes even on the same machine. Register the new system on the portal and obtain codes before the cut-over - the registration is confirmed by Xi Software rather than granted on the spot.

The saved state. The job, variable and user files in the spool directory are raw memory images in the machine's own byte order and word size, with no marker to say so and no validation on load. Copying them to a machine of a different byte order or word size gives a queue full of nonsense rather than an error. Use the conversion tools on the source machine and carry their text output; the conversion tools apply plausibility checks and will usually refuse a file from a foreign machine with "I am confused about the format of your job file", which is a refusal to respect rather than override.

The model behind both is set out in the concept article on what travels between machines; the export and restore procedure is in the Xi-Batch migration article.

Performance Characteristics

Throughput is dominated by the storage holding the spool directory and by the work the jobs themselves do, rather than by the platform. Two things are worth attending to:

• Local storage always outperforms network storage for spool directories, and network storage brings the locking problems described above as well.
• Shared memory is sized once at startup, from the job and variable counts on the start command. Growing a segment while running is supported but costs a copy, so size it generously for the site's busiest day.

Where a specific platform needs tuning, that will show up as a failure to allocate shared memory at startup rather than as gradual slowness.

Best Practices for Multi-Platform Environments

Consistent directory structure

Use the default locations on every platform unless there is a reason not to. Where you do relocate, record the base directory alongside the installation.

Consistent account names

The generated restore scripts name users and groups by name, so what matters is that every account exists on the target. The numeric ids need not match, and by default they will not.

Consistent interpreter names

Keep the interpreter names the same on every platform and let the paths differ. A job names the interpreter by name, so the same job then runs everywhere and only the table changes.

Network configuration

Maintain one master list of the hosts in the installation and distribute it, prepending each machine's own localaddress line. The files are not byte-identical across machines.

Platform documentation

Record the platform-specific settings for each host - interpreter paths, base directories, the startup mechanism in use.

Testing migration scripts

Replay the generated scripts on a non-production machine of the target platform before the real cut-over.

Scheduled maintenance windows

Stop the product before exporting. The saved files are rewritten only every few minutes, so an export taken from a running system can be several minutes behind the queue.

Rollback procedures

Keep the old system available, and its licence intact, during the parallel operation phase.

Platform Change Checklists

From Solaris to Linux:

• Correct interpreter paths in the interpreter table with btcichange, before starting the scheduler
• Replace the init.d script with the packaged systemd unit
• Verify terminfo entries exist for the terminal types in use
• Register the new machine on the portal and obtain new licence codes

From HP-UX to Solaris:

• Correct interpreter paths in the interpreter table with btcichange
• Check the shared-memory limit on the target against the queue size you intend to start with
• Confirm the spool directory is on a local file system
• Add the startup script by hand or let the installer write the System V one
• Register the new machine on the portal and obtain new licence codes

From AIX to Linux:

• Correct interpreter paths in the interpreter table with btcichange
• Replace the /etc/inittab entry with the packaged systemd unit
• Check that any job script relying on Korn shell behaviour still behaves under the interpreter it now names
• Register the new machine on the portal and obtain new licence codes

Each platform pair has its own considerations. Test the whole migration on a non-production machine of the target platform before doing it for real.

WARNING: a move to another machine always needs new licence codes

The licence is keyed to the machine's hardware address, so it does not validate on different hardware and cannot be carried across by copying the file. Register the new system on the portal and obtain codes before the cut-over. A trial licence covers the interval if the confirmation has not yet come through.

WARNING: restore the interpreter table before the scheduler starts

Any saved job naming an interpreter that is not in the table when the job file is loaded is silently reset to the first entry, and correcting the table afterwards does not put it back.

WARNING: spool directories must be on local file systems

Do not place the spool directory (/var/spool/xi/batch, or wherever the installation put it) on NFS or any other network file system. Advisory record locking over the network is unreliable and the performance is poor.

WARNING: do not copy the spool directory between machines of different architecture

The saved job, variable and user files are raw memory images in the machine's own byte order and word size, with nothing in them to say so. Use the conversion tools on the source machine and carry their text output.