Pipelines and provenance

A stage’s key is the SHA-256 of its type, its resolved inputs, the content digest of a case file named by case, the keys of the stages it depends on and the code state. If the stored record of a stage has the same key and succeeded, the stage is skipped and its outputs reused. Changing an input or an upstream stage therefore invalidates exactly the stages that depend on it. --force ignores the cache; cache: false marks stages that must always run (deployment). A stage whose execution fails replaces an earlier success record, so it is not served from the cache afterwards.

The code state is the commit for a clean working tree and <commit>+dirty.<digest> otherwise, where the digest covers git diff HEAD and the untracked, non-ignored files: every uncommitted edit, and every new file that git does not ignore, invalidates every stage. Three paths of the Gyre checkout are left out, because no stage reads them: uncommitted changes under devlog/ and .github/ and to CLAUDE.md do not change the code state (a commit still does). Every other file counts, README.md included. Where git cannot describe the tree Gyre runs from (an installed package, the container image, no git) the code state is the revision followed by a hash of the package sources, and docs and verify stages always run, because they read files outside the package.

A shell stage reads the tree of the directory it runs in (with.cwd, else the engine’s working directory). If that directory is not inside the Gyre checkout, the state of the git work tree around it enters the stage key as well; if there is no git work tree there, the stage is never served from the cache.

The key is formed before a stage runs. If the code state has changed by the time a compute or shell stage ends, nobody knows which code the stage saw: its record gets code_changed: true and revision_after, its stage.json is stored with key: null so that it is never served from the cache, the engine prints the code changed while the stage ran (...); its result is not cached, and the stages that follow are keyed on the new state.

Runs that share <workdir>/<name> are serialised by an advisory file lock on that directory (POSIX; without fcntl runs are not serialised): a second run waits for the first, then finds its results in the cache where the keys match.

A failed stage (non-zero exit, exception, or a gate whose check fails) marks the pipeline failed; stages depending on it are recorded as skipped with the reason; independent stages still run. The pipeline record (pipeline.json) lists every stage with status, key, wall time and outputs.

The documentation build reads verification files and renders their provenance next to each chart and table. A deployment builds its site from the working tree it runs, so the served documentation, the served API and the published evidence refer to one revision.

+dirty on a stamp means that code a run can depend on differed from the commit. Three kinds of path do not count: the effort ledger and roadmap under devlog/, the hosted workflow definition and the agent instructions, which no code reads; and the verification records under var/verification/, which runs write and no computation reads. Without the last exception a verification run would be dirty by its own output: the first study to finish rewrites its tracked record, and every later study of the same run would carry +dirty on a clean checkout. The records are committed after the run, so a record names the commit of the code that produced it, which is the parent of the commit that stores it.

Next to every chart and table the site compares the revision of the record with the revision it was built from. A record is marked with a tick when it was produced by the build’s own commit, by its parent, or by an earlier commit with the same code: the build walks back from its commit through the commits that change only documentation text (docs/content, the README), evidence records or bookkeeping, up to and including the last commit that changed anything else (gyre.identity.code_base), and a record from any of them was produced by the code of the build. A record from any other commit, or from a tree with uncommitted changes, is marked as a mismatch. The theme and the builder of the documentation count as code.

The whole pipeline was last run with make ci on the development host (8 CPU threads, RTX 3060 Ti) at revision 76a489be21, from a clean working tree, started 2026-11-11 10:38:

58 minutes in all. verification-short (six cheap studies) and lint form the fast tier (make ci-fast); verification is the whole suite, whose records this site renders. On that host the image stage needs GYRE_BUILD_NETWORK=host (why).

The CI pipeline is an ordinary pipeline, runnable locally with make ci; a hosted workflow (.github/workflows/ci.yml) mirrors it without the deployment. Because of its shell stages, ci is not startable through the service unless the operator sets GYRE_PIPELINES_ALLOW_SHELL=1. The final stage calls tools/deploy.py up, which refuses a port on which something already listens, builds the site, starts the service on all interfaces, and runs deploy/smoke.py against the tailnet address: health, deployment info, documentation pages, presets, structured errors, pipeline definitions, the MCP tool list, a real solver job with an adjoint gradient (never answered from the result cache), and metrics. deploy.py smoke-tests and records only the process it started: an answer from another service on the port is an error.

• Native (tools/deploy.py up|down|status): named deployments under var/deploy/<name>/, each with its own site, data directory, log and record.
• Container (deploy/Dockerfile, deploy/compose.yaml): multi-stage image that installs the package, stamps the revision, builds the site, ships the analysis pipeline definitions and the examples, and serves on port 8600 bound to all interfaces; GPU and AVX-only hosts are handled by build arguments.
• systemd (tools/deploy.py systemd, from the template deploy/gyre.service.in): a user unit for this checkout that builds the documentation before every start and restarts the service on failure.