Diagnosing Xi-Batch Variables That Do Not Reach Another Host

How Xi-Batch Shares Variables Between Hosts

Synchronisation is the wrong word for what the product does, and most of the confusion this article exists to clear up comes from expecting it. Every variable has exactly one owning host - the machine it was created on. That host holds the master copy. Other hosts hold a copy that the owner pushes to them. There is no reconciliation between copies, no acknowledgement, no retry and no periodic refresh.

A variable's export state decides who gets a copy. There are three states, not two.

Cluster implies exported. Setting the cluster marker on a variable that is not exported is refused with "Cannot set NAME local and clustered". A cluster name must be unique on each machine - two users cannot each have one of that name - and a clustered variable cannot be renamed.

The three states are what the variable log records when the flag is changed, as the words Cluster, Exported and Local only. See article 18 for the variable log, and article 31 for variables in general.

How a Change Reaches the Other Hosts

For an ordinary change made with btvar or btq:

1. The change is applied to the master copy on the owning host.
2. As part of applying it, the owning host writes the new value to every host it is currently connected to, in one pass.
3. Each receiving host stores it in its replica. Nothing is sent back and nothing is acknowledged.

A change made against another host's variable is not applied locally at all. Running btvar on host B against hostA:COUNTER forwards the request to hostA, which applies it and then broadcasts the result to everyone, including B. A replica can never be written locally, and if the owning host is not connected the request fails at once with "Specified host is off line".

Three consequences a reader needs before diagnosing anything:

• Nothing polls. If a value looks old, waiting will not refresh it. Either the change never reached this host or it never happened.
• There is no catch-up. A host that was disconnected when a change was made never receives that change. It receives the variable's current value when a connection is next established, and nothing in between.
• A change made by a job is propagated differently. Rather than sending a value, the host that ran the job tells every other host to carry out the same assignment list itself. Every host that holds the job arrives at the same answer independently. A host that was disconnected at that moment does not do so, and keeps the old value until a connection sync replaces it.

Bulk transfer when a connection is established

When two schedulers connect - at startup, or when an operator runs btconn - each sends the other every exported variable it owns, together with its exported jobs. This is the only mechanism that brings a stale replica up to date. It is deliberately throttled: the scheduler pauses for the number of seconds in the MSGTXDELAY setting of /etc/xi/batchconfig after every MSGTXSIZE messages (defaults 2 seconds and 20 messages), so a large exported set takes a noticeable time to transfer.

Simultaneous changes on two hosts

The last change applied wins, and neither party is told. There is no lock on a variable and no conflict detection across hosts. The owning host applies requests one at a time in the order they arrive; the value everyone ends up with is the last one applied.

There is one guard, and it only covers one case. A request made on the owning host itself carries the change sequence number the requester read, and is refused if the variable has been changed since:

Some other user attempted to update variable COUNTER
at the same time. Please check and perhaps try again.

That check is applied only to requests originating on the owning host. A request arriving from another host is not sequence-checked - the sequence number is not even sent over the network. So a read-modify-write done as two commands (read the value, compute, set it) from two different hosts at the same moment silently loses one of the updates, with no error and no log entry.

Where a counter must be correct, do the read and the write on the variable's own host, or give each host its own counter with a cluster variable and add the values up afterwards.

What Happens When a Host Becomes Unreachable

This is the behaviour most likely to surprise. A peer is treated as gone when its scheduler shuts down cleanly, or when a read or write on the connection fails. Nothing times a peer out for silence: the keepalive is sent only after twice the per-host timeout has passed with no traffic (default 1000 seconds per host, so about 33 minutes), and an idle connection to a machine that has vanished may not be noticed until then.

When the local scheduler does decide a peer has gone, it goes through its variable table and, for every variable owned by that peer:

• if no local job refers to it, the variable is deleted. It disappears from btvlist entirely.
• if a local job refers to it in a condition or an assignment, it is kept as a placeholder: the name and ownership remain, the permissions are set to nothing, and the value can no longer be read.

Neither of those is written to the variable log or to the scheduler report file. The variables of a host that has gone away simply stop being there.

Jobs are then affected in three different ways, and the first is the dangerous one.

• A condition or assignment on such a placeholder that is not marked critical is skipped. The job is treated as though the condition were satisfied, and it runs. This is the default: a condition is not critical unless it was created as one.
• A condition or assignment marked critical holds the job indefinitely. The scheduler passes over it on every scan. The job's progress state does not change, no message is written anywhere, and there is no timeout. It looks exactly like a job that is simply waiting for its time.
• A job of yours that was running on the departed host is marked Abrt and appears in the job log with the event network-aborted. Its start-of-job assignments have already been made and its end-of-job assignments never will be.

The hold is released when the host returns and the real variable arrives. One thing to know about that moment: if the returning variable's permissions no longer allow the job's owner the access the job needs, the condition or assignment is deleted from the job without a message, and the job then runs without it.

Marking a condition critical

btr -k marks the conditions given after it critical, and -K marks them not critical; -b and -B do the same for assignments. Each applies to the -c or -s options that follow it on the command line.

# Do not run this job at all if server2 is unreachable
btr -k -c 'server2:BACKUP_DONE=Yes' -h "Nightly report" report.sh

The marker only takes effect on a variable written with a host prefix. On a plain local variable name it is accepted and discarded.

Diagnostic Approach

Step 1: Confirm this host is running networked at all

Networking is part of the licence. Where it is not licensed the scheduler starts normally and runs local jobs with no error, and no variable can be exported. The symptom is that an export flag will not stay set.

Two checks, and they answer different questions. The licence itself:

xb-checklic

A networked licence prints the line "Validated for networks" among its output. If that line is absent, the licence does not cover networking.

Then whether the running scheduler is networked, which is the MACHINE system variable - it is created only when networking is available:

btvlist -F '%N %V' | grep '^MACHINE'

If MACHINE is absent, this scheduler is not networked and nothing else in this article applies. The exported and cluster markers are stripped from every variable as the saved variable file is read at startup, and a later btvar -E reports success while leaving the flag off. Re-licensing the machine is the fix, and every variable that was exported has to be exported again afterwards.

Step 2: Check the variable's export state

The default btvlist format shows the export column but not the cluster column, so an exported and a clustered variable look identical in a plain listing. Ask for both:

# Name, value, export marker, cluster marker
btvlist -H -F '%N %V %E %K'

The export column shows the word Export or nothing; the cluster column shows Cluster or nothing. Remote hosts' variables are included by default and are shown with a host: prefix on the name; btvlist -L limits the listing to local ones.

To set the state:

# Exported: one value, held here, pushed to the other hosts
btvar -E -k COUNTER

# Cluster: every host has its own COUNTER
btvar -E -K COUNTER

# Local only
btvar -L COUNTER

There is no btvar -e and no btvar -v. Reading a variable takes no option at all - btvar COUNTER prints the value and nothing else.

Step 3: Check the hosts file

The hosts file is /etc/xi/batch-hosts. Older installations had /etc/Xibatch-hosts, and that name still appears in some of the product's own messages and in the manuals; the file the scheduler reads is the first one.

One host per line. Fields are separated by spaces or tabs, in this order: host name or IP address, alias, flags, timeout in seconds. Blank lines and lines beginning with # are ignored. A line may carry the host name alone. An alias of - means no alias, and a line giving a dotted IP address must supply an alias. Flags are a comma-separated list; the two that matter here are:

• probe - find the host with a UDP probe before connecting to it.
• manual - do not connect to this host automatically. It is connected only when an operator runs btconn.

If the file contains a localaddress line, giving the address this machine should present to the others, it must be the first line that is not blank or a comment. Anywhere else it is discarded and the file is recorded as containing errors. Because each machine needs its own localaddress line, the files on the different hosts are not copies of one another.

Errors in the file are reported once, when the scheduler starts, in the scheduler report file:

btsched: Warning: There are errors in the /etc/Xibatch-hosts file

A hosts file that is missing altogether produces no message at all - the scheduler simply connects to nothing.

The scheduler reads this file once, at startup. An edit has no effect until btsched is restarted, except that btconn will connect to a single host named in the edited file without one.

Article 41 covers setting the file up; this article assumes it already exists.

Step 4: Check that the two schedulers are connected

The connectivity check is btconn:

btconn server2

btconn asks the local scheduler to connect to that host now. It takes exactly one host name and no options, needs the stop scheduler privilege, prints nothing on success and returns exit status 0; on failure it prints "btconn - connection to server2 failed" and returns a non-zero status. btdisconn is the reverse.

ping and ssh are not the test. They exercise the network and the login path, and a host can answer both while having no Xi-Batch connection at all - because it is flagged manual, because its scheduler is stopped, or because the connection was lost and never retried.

That last point is worth stating on its own. A host connects to its peers once, at scheduler startup, and never retries. A peer that was down at that moment is not connected to again by this machine. Recovery depends on the other end starting up and connecting inwards, or on an operator running btconn. In a pair of machines that were rebooted in the wrong order this is the whole of the fault.

The schedulers talk over the service entries installed with the product. Confirm they are present on both machines:

getent services xibatch
getent services btq

xibatch 2050/tcp        # Scheduler-to-scheduler connection
xibatch 2050/udp        # Probe, for hosts flagged probe
btq     2150/tcp        # Feeder port used by btq and the listing commands

A firewall between the two machines must permit TCP 2050 in both directions, and UDP 2050 as well where the probe flag is used. On a networked host the xibatch entry is not optional: without it the scheduler stops during startup rather than running without networking.

Step 5: Compare the value on each host

btvar server2:BACKUP_STATUS

This reads the local copy. btvar attaches to the local scheduler's shared memory; it does not contact server2. So it tells you what this host believes, and it cannot tell you whether server2 is reachable. Two failures look alike and mean different things:

• "btvar: Unknown host name in 'server2:BACKUP_STATUS'." - server2 is not in the hosts file and is not resolvable. This is a naming fault, not a connectivity one.
• "btvar: Unknown variable server2:BACKUP_STATUS" - this host has no copy. Either the variable was never exported, or server2 went away and the copy was discarded.

To find out what the owning host holds, run btvar on the owning host. Comparing the two answers is the diagnosis:

# On the owner
btvar BACKUP_STATUS

# On the other machine
btvar server2:BACKUP_STATUS

Same value: the variable is propagating. Different values: the two schedulers are not connected, or were not connected when the change was made. Absent on the second machine: it is not exported, or the connection has been lost since.

Step 6: Read the scheduler report file - and know what is not in it

tail -50 /var/spool/xi/batch/btsched_reps

What is recorded there: the hosts-file warning at startup, scheduler start and stop, probe messages from an unexpected address, a reconnection to a host that was already connected, and fatal errors.

What is not recorded there, and this is the trap: nothing about variables at all. A value arriving from another host is applied with no entry - not in this file and not in the LOGVARS variable log. A host connecting or disconnecting is not recorded. A dead host's variables being discarded is not recorded. Finding nothing here about a variable is the normal case and tells you nothing about whether the network is working. Use btconn and the value comparison in step 5 instead.

Resolving Common Issues

Issue 1: The variable is not exported

Symptom: the variable is listed on its own host and is absent everywhere else.

# On the owning host
btvar STATUS
Ready

# On the other host
btvar hostA:STATUS
btvar: Unknown variable hostA:STATUS

Solution:

# On the owning host
btvar -E -k STATUS

# Confirm
btvlist -H -F '%N %V %E %K' | grep STATUS

The export flag can only be changed on the owning host; on any other machine the attempt is refused. Setting it broadcasts the variable to the connected hosts at once, so it should appear on the other machine immediately.

Where the flag will not stay set, go back to step 1: an unlicensed network strips it silently.

Issue 2: The two schedulers are not connected

Symptom: the other host's variables have disappeared from btvlist, or an attempt to set one fails with "Specified host is off line".

Solution:

# Ask the scheduler to connect now
btconn server2

# Then check the value arrived
btvar server2:BACKUP_STATUS

If btconn fails, check in this order: the scheduler is running on server2; server2 is in /etc/xi/batch-hosts on this machine and this machine is in the file on server2; the xibatch service entry exists on both; TCP 2050 is open between them.

Where server2 is flagged manual in the hosts file, btconn after every restart is the intended way to use it and there is nothing to fix.

Issue 3: The value on the other host is old

Symptom: the owning host shows a new value, another host still shows the old one.

There is no cache to flush and nothing to expire. One of two things has happened: the two hosts were not connected when the change was made, or they are not connected now.

Solution:

# On the host with the old value
btconn hostA
btvar hostA:STATUS

Establishing the connection transfers the current value, which is why btconn fixes this and waiting does not. Restarting a scheduler also works, because it connects on startup, but it is a much larger action for the same effect and it stops every job the scheduler is running.

Issue 4: Two hosts update the same variable and one update is lost

Symptom: a counter incremented by jobs on two machines advances by one where it should have advanced by two.

The cause is the read-modify-write, not the network. Two shell scripts that each read the value, add one and write it back will overlap, and the cross-host case is not detected at all.

Solution: let the scheduler do the arithmetic. A job assignment using += is applied by the scheduler as a single operation rather than as a read followed by a write:

btr -h "Counted job" -f S -s 'counter+=1' work.sh

Here -f S makes the assignment happen at job start; without -f an assignment is made at start and reversed at the end of the run. Article 33 covers conditions and assignments in full.

Where a genuine mutual exclusion is needed, use a condition and an assignment on the same job rather than a shell loop, so that the scheduler tests and sets in one step.

Issue 5: A job ran when its remote condition should have stopped it

Symptom: server2 was down, and a job whose condition tests a variable on server2 ran anyway.

This is the default behaviour and it is the most damaging item in this article. When server2 went away, the condition's variable became an unreadable placeholder, and a condition on a placeholder that is not marked critical is skipped - the job is treated as though the condition were met.

Solution: mark the condition critical. Criticality is not a property that can be toggled on an existing condition; the condition list has to be restated:

# Replace the job's conditions with the same one, marked critical
btjchange -y -k -c 'server2:BACKUP_DONE=Yes' 4207

-y clears the existing conditions and -k marks the ones that follow as critical. Check the result with btjlist -F '%N %C', which prints the conditions in full.

The job will then wait rather than run. Decide deliberately which of the two you want: waiting is safe for a job that must not run on stale data, and unsafe for one whose whole purpose is to run every night.

Issue 6: A job with a critical condition never runs

Symptom: the job sits in the queue with an unremarkable state and never starts, while other jobs run normally.

A critical condition on a variable belonging to an unreachable host holds the job with no timeout, no message and no visible change of state. There is no command that shows this state directly, so it has to be inferred: list the job's conditions, look for a host prefix, and check whether that host is connected.

# What does the job depend on?
btjlist -H -F '%N %H %C' | grep 4207

# Is the named host's copy still there?
btvar server2:BACKUP_DONE

A placeholder has its permissions set to nothing, so btvar reports it as an unknown variable exactly as a deleted one would. Either answer means this host no longer holds a usable copy, which is what a held job looks like from the outside. Confirm with btconn: if it connects, the host was gone.

Solution: bring the host back and connect it, which releases the job:

btconn server2

Where the host is not coming back, either restate the condition without -k so the job runs regardless, or remove the condition:

# Drop the conditions entirely
btjchange -y 4207

Network-Wide Variable Management

List the exported and clustered variables

btvlist -H -F '%N %U %V %E %K'

This lists every variable this host knows about, its own and its replicas of other hosts'. The name carries a host: prefix where the variable belongs to another machine.

List one host's variables

# Just this host's own
btvlist -L -H -F '%N %V %E %K'

# Just server2's, as this host sees them
btvlist -N -F '%N %V %E %K' | grep '^server2:'

Remember that the second of these shows this host's replicas, which are only as fresh as the last connection.

Tracking changes

The LOGVARS variable log records changes that a host applies as a result of a local or forwarded request. It does not record a value arriving from another host, so the log on the receiving machine stays silent while the value changes underneath it. A network-wide picture cannot be assembled from one host's log; enable LOGVARS on the owning host, where every change to that variable is applied. Article 18 covers the log in full.

Practices That Avoid the Failure Modes Above

Keep the number of exported variables small

every one of them is sent to every host on every connection, and the transfer is deliberately throttled.

Decide criticality deliberately on every remote condition

the default is that the job runs when the other host is unreachable. State which behaviour you want rather than accepting the default by omission.

Give the arithmetic to the scheduler

use a job assignment with += rather than reading and writing the variable from a script, and the lost-update case does not arise.

Where each host needs its own value, use a cluster variable

it is the mechanism built for that, and it removes the contention entirely.

Run btconn from a job or a startup script

since a scheduler never retries a failed connection, a periodic btconn against each peer is the practical way to recover from a peer that was down at start of day. It costs nothing when the host is already connected.

Change the hosts file and restart together

the scheduler reads the file once. An edit that is never followed by a restart is a fault waiting for the next reboot to appear.

Troubleshooting Checklist

When a variable is not reaching another host:

• The MACHINE system variable exists on both hosts (networking is licensed)
• The variable is marked Export on its owning host
• Both hosts appear in /etc/xi/batch-hosts on the other, with localaddress first if present
• The scheduler has been restarted since the hosts file was last edited
• btconn to the other host succeeds
• The xibatch and btq service entries exist on both hosts
• TCP 2050 is open between the two machines
• The other host is not flagged manual without something to run btconn
• The variable name is spelled with the owning host's name in the prefix, not the reader's own

Verification After Changes

# On the owning host
btvar -C -E -k -s "checked $(date '+%H:%M')" SYNC_TEST
btvar SYNC_TEST

# On the other host, immediately
btvar hostA:SYNC_TEST

The value should be identical straight away: the change is written to the connected hosts as part of applying it, so there is no interval to wait out. If the second command shows the previous value, the two schedulers are not connected - run btconn and try again.

Note the argument order: btvar -s takes the value, and the variable name is the last argument.