How the products are tested
Nine stages between the first line of code and a package you can download. A failure at any of them stops the change there.
Software that runs unattended overnight is judged on the night it goes wrong. That is the whole argument for testing it harder than anyone asked, and it is why every stage below is automatic except the two where a person has to think.
The order a change goes through
| Stage | What it means | |
|---|---|---|
| 1 | A failing test comes first | Every change starts with a test that reproduces the fault or specifies the behaviour, run where it fails. The change is made, the same test passes, and both results are recorded with it. |
| 2 | Two people review it | Two people, independently of whoever made the change. The first takes it commit by commit and runs the gate; the second accepts or rejects it. A rejection sends it back to stage 1. |
| 3 | The build system checks itself | The toolchain has its own test suite, and all of it runs before any change to the way the products are built is accepted. |
| 4 | The products are built and exercised | Both products are built from source and put through families of test covering units, integration, capacity, burst, security, hardening - and the reproduction of every fault ever reported. |
| 5 | Every platform is built clean | Each platform is built in its own container from a pinned image, and the build record names the source, the toolchain and the image - so any package can be traced to exactly what built it. |
| 6 | Every package is verified | Six independent checks on every package for every platform, in a fresh system. |
| 7 | The tar is tested as a customer uses it | The tar you unpack is installed by its own installer, answered question by question, then upgraded over and removed. |
| 8 | The code is fuzzed | Deliberately malformed input is thrown at the schedulers, the spoolers and the API before every release. |
| 9 | The release ships the tested bytes | Nothing is rebuilt for release. The verified binaries are the ones packaged, so what you install is what was tested. |
What every package has to pass
Stage 6 is where most of the work is. Each of these runs on every package, for every platform, in a clean system:
It is complete
Every program that should be in the package is in it, runs, and reports the version it claims to be.
It is hardened
Every binary is built with the exploit mitigations the platform offers, and each one is checked rather than assumed.
It is ours
The package carries a signature from our release key, checked against the key we publish.
It installs and uninstalls
It installs through the platform's own package manager with its dependencies resolved, the service comes up, the configuration and licence are in place - and removing it leaves the system clean.
It upgrades without loss
Upgrading from earlier releases is proven rather than assumed: the service, configuration, licence and spool are all shown to survive.
It works
A job is submitted and followed through the scheduler or the spooler, on the installed package, on that platform.
Fuzzing
Before any release, deliberately malformed input is thrown at the parts of the products that read from the network: the schedulers' and spoolers' job and variable handling, and the programming interface. The targets are built with the address and undefined-behaviour sanitisers, so a fault that would otherwise pass unnoticed is caught as it happens.
The result is recorded against the exact source it was run on, and a release of that source is refused until the record is clean.
Faults are met here rather than at two in the morning on somebody's month end. That is what the process is for.