---
title: Pipelines
summary: Chain analyses into reproducible workflows with a small YAML vocabulary. Stages are cached by content, so re-running a pipeline recomputes only what changed.
section: guide
order: 7
level: 2
keywords: [pipeline, YAML, sweep, gates, cache]
---
<!-- gyre docs: expanded Markdown of guide/pipelines.md (generated tables, verification numbers and chart data are written out). Base level of this page: L2 Practice; passages at another level are bracketed by 'begin level N' / 'end level N' comments and start with [LN Name]. Built from revision ceb1f4b0be. -->

```bash
gyre pipeline run pipelines/airfoil-characterisation.yaml
gyre pipeline run pipelines/airfoil-characterisation.yaml --force     # ignore the cache
gyre pipeline run pipelines/airfoil-characterisation.yaml --param NAME=VALUE --record FILE
gyre pipeline run pipelines/airfoil-characterisation.yaml --param-json 'NAME=JSON'
```

`--param` overrides a pipeline parameter: the value is read as JSON, otherwise as text, and a
number or boolean given for a parameter whose default is text stays text. `--param-json` (also
repeatable) takes a value that must be valid JSON and uses it exactly as given, whatever the
declared default is; it wins over `--param` for the same name. `--record` also writes this
run's record to `FILE`.

Pipelines can also be started through the service (`POST /api/v1/pipelines/{name}/runs`) and
by agents (`run_pipeline`, `get_pipeline_run`). The definitions of a deployment are listed by
`GET /api/v1/pipelines`, each with `runnable`: a definition with `shell` stages cannot be
started through the service or MCP unless the operator sets `GYRE_PIPELINES_ALLOW_SHELL=1`.
Runs of the same pipeline in the same work directory execute one after another.

<!-- begin level 3 (Mechanics) -->
**[L3 Mechanics] Runs through the service**

The service lists the `*.yaml` files of the directory named by `GYRE_PIPELINES_DIR` (default:
`pipelines` in the service's working directory). The container image ships the analysis
definitions in `/app/pipelines`; the CI pipeline is not part of the image.

A run started through the service takes a stage's `case` as a preset name or a CaseSpec
object. The check is made on every stage's `case` after the run's parameters are substituted,
whatever the parameter is called: a string that is not a preset name, or a `case` that cannot
be determined before the run (a reference to another stage's output), is refused with 404
`UnknownPreset`. The stages of such a run also refuse to read a case file
(`GYRE_PIPELINE_CASE_FILES=0` in their environment). Paths work with `gyre pipeline run` on the
host. A run leads its own process group and is terminated, with its stage processes, when the
service shuts down.
A definition that is not `runnable` answers 403 `PipelineNotRunnable`, with the reason as the
`hint`. A string parameter stays a string (`"123"` is not turned into a number). The record of
a run is the `record` of its status document (`GET /api/v1/pipelines/runs/{id}`).

<!-- end level 3 -->

## A complete example

```yaml
name: airfoil-characterisation
params:
  case: naca0012_euler
stages:
  - id: baseline
    uses: solve
    with:
      case: "${params.case}"
      gradients: {of: [cl, cd], wrt: [alpha, mach]}
      error_estimate: [cd]

  - id: polar
    uses: sweep
    with:
      case: "${params.case}"
      parameter: alpha_deg
      values: [0.0, 1.0, 2.0, 3.0, 4.0]
      outputs: [cl, cd, cm]
      gradients_wrt: [alpha]

  - id: gates
    uses: assert
    with:
      checks:
        - {name: lift at 2 deg, value: "${stages.baseline.outputs.outputs.cl}", min: 0.25, max: 0.32}
        - {name: polar converged, value: "${stages.polar.outputs.all_converged}", equals: true}

  - id: summary
    uses: report
    needs: [gates]
    with:
      title: NACA 0012 characterisation
      values:
        cl: "${stages.baseline.outputs.outputs.cl}"
        dcl_dalpha: "${stages.baseline.outputs.gradients.cl.alpha}"
```

<!-- begin level 2 (Practice) -->
**[L2 Practice] Vocabulary**

| Key | Meaning |
|---|---|
| `params` | pipeline-level values, referenced as `${params.name}` |
| `stages[].id` | unique name; outputs are referenced as `${stages.<id>.outputs.<path>}` |
| `stages[].uses` | `solve`, `sweep`, `order_study`, `verify`, `docs`, `shell`, `assert`, `report` |
| `stages[].with` | inputs of the stage |
| `stages[].needs` | explicit dependencies (references add implicit ones) |
| `stages[].device` | `auto`, `cpu` or `gpu` for compute stages |
| `stages[].cache` | `false` to always run |

A reference that is the whole value keeps its type (number, boolean, list); inside a longer
string it is formatted as text.

An `assert` stage takes a non-empty `checks` list. A check is `{value, name}` (the name is
optional) with at least one of the criteria `min`, `max`, `abs_max` and `equals`. Unknown keys,
a check without a criterion and a missing or empty `checks` list are rejected when the pipeline
is loaded, before any stage runs. `min`, `max` and `abs_max` compare numbers: a value that is
not a number (`null` from a diverged solve, a string, a list) or a criterion that is not a
number fails the check, and the check result says why in `reason`. `equals` compares values of
any type. A failed gate ends the stage with `PipelineError: gate failed: […]`.

A `solve` stage takes `case`, `overrides`, `gradients`, `error_estimate`,
`require_converged` (default true: the stage fails if the solve does not converge) and
`adapt`:

```yaml
  - id: accurate
    uses: solve
    with:
      case: bump_euler
      overrides: {discretization.order: 1}
      adapt: {output: cd, tolerance: 1.0e-4, max_order: 4}
  - id: accuracy-gate
    uses: assert
    with:
      checks:
        - {name: tolerance met, value: "${stages.accurate.outputs.adaptation.met}", equals: true}
```

`adapt` selects the polynomial degree by the estimated error of `output`
(`max_order` defaults to 5). Unknown keys, a missing `output` or `tolerance` and a value that
is not a mapping are errors before the solve. The stage does not fail when the solve converged
but the tolerance was not met at `max_order`; `adaptation.met` is then false, which is what the
gate above tests. `met` is true only for a finite estimate whose adjoint solve converged, and
the selection ends at the first degree that does not converge; the level records in
`adaptation.levels` are described under [Service](service.md#jobs-one-shot-analyses).

<!-- end level 2 -->

<!-- begin level 2 (Practice) -->
**[L2 Practice] Stage inputs: the `with:` keys of every stage type**

Keys marked *required* have no default; a stage without them fails when it runs. `case` is a
preset name, the path of a specification file, or an inline specification (through the
service: a preset name or an inline specification only). `overrides` is a mapping of dotted
paths, as with `gyre run --set`.

**`solve`** — one analysis (the files of `gyre run -o` are written to the stage directory).

| Key | Type | Default | Meaning |
|---|---|---|---|
| `case` | name, path or mapping | *required* | the case to solve |
| `overrides` | mapping | `{}` | changes to the case |
| `gradients` | `{of: [...], wrt: [...]}` | none | total derivatives of scalar outputs; `wrt` accepts `alpha_deg` |
| `error_estimate` | list of output names | none | adjoint-based error estimates |
| `require_converged` | boolean | `true` | fail the stage if the solve does not converge |
| `adapt` | `{output, tolerance, max_order}` | none | order selection, see below; `max_order` defaults to 5 |

**`sweep`** — one warm-started solve per value, in one solver process.

| Key | Type | Default | Meaning |
|---|---|---|---|
| `case` | name, path or mapping | *required* | the case |
| `overrides` | mapping | `{}` | changes to the case |
| `parameter` | input name | *required* | the swept input: a flow input of the discipline (`mach`, `alpha`, …) or `alpha_deg` |
| `values` | list of numbers | *required* | the values, in the order they are solved |
| `outputs` | list of output names | `[cl, cd, cm]` | outputs recorded per row; may include `f_surface` |
| `gradients_wrt` | list of input names | none | adds `d<output>/d<input>` columns for the scalar outputs |

**`order_study`** — the same case at several polynomial degrees, each solved from scratch.

| Key | Type | Default | Meaning |
|---|---|---|---|
| `case` | name, path or mapping | *required* | the case |
| `orders` | list of integers | *required* | polynomial degrees; each replaces `discretization.order` |
| `overrides` | mapping | `{}` | changes to the case (applied before the degree) |
| `error_estimate` | list of output names | none | adds `<output>_corrected` and `<output>_error_estimate` to each row |

**`verify`** — the verification suite.

| Key | Type | Default | Meaning |
|---|---|---|---|
| `suite` | `quick` or `full` | `quick` | the level of the studies |
| `output` | directory | the stage directory | where the study records and `index.json` are written |
| `only` | list of study ids | all studies | run a subset |
| `strict` | boolean | `true` | fail the stage if a study of this invocation fails |

**`docs`** — build the documentation site.

| Key | Type | Default | Meaning |
|---|---|---|---|
| `output` | directory | `site` | where the site is written |
| `verification` | directory | `var/verification` | the study records the pages render |

**`shell`** — a command on the host (not startable through the service unless the operator
sets `GYRE_PIPELINES_ALLOW_SHELL=1`).

| Key | Type | Default | Meaning |
|---|---|---|---|
| `run` | string | *required* | the command line, run by the shell; a non-zero exit code fails the stage |
| `cwd` | directory | the directory `gyre pipeline run` was started in | working directory of the command |
| `timeout` | number, seconds | none | the command and everything it started are stopped, and the stage fails, when it runs longer |

**`assert`** — a gate.

| Key | Type | Default | Meaning |
|---|---|---|---|
| `checks` | non-empty list | *required* | each `{value, name?, min?, max?, abs_max?, equals?}` with at least one criterion; rules below |

**`report`** — a Markdown and JSON summary (`report.md`, `report.json` in the stage directory).

| Key | Type | Default | Meaning |
|---|---|---|---|
| `title` | string | `Pipeline report` | heading of the report |
| `values` | mapping name → value | `{}` | the rows, usually references to outputs of other stages |

```yaml
  - id: degrees
    uses: order_study
    with: {case: bump_euler, orders: [1, 2, 3], error_estimate: [cd]}
  - id: studies
    uses: verify
    with: {suite: quick, only: [free_stream, entropy_conservation], output: var/verification}
  - id: site
    uses: docs
    needs: [studies]
    with: {output: site, verification: var/verification}
  - id: lint
    uses: shell
    with: {run: ".venv/bin/ruff check src tests", timeout: 600}
```

<!-- end level 2 -->

<!-- begin level 3 (Mechanics) -->
**[L3 Mechanics] What each stage returns**

| Stage | Outputs |
|---|---|
| `solve` | `converged`, `outputs`, `residual_norm`, `timing`, `case`, optional `gradients`, `error_estimate`; with `adapt` also `adaptation` (`output`, `tolerance`, `met`, `order`, `value`, `corrected`, `error`, `levels`), and `case` describes the final degree |
| `sweep` | `parameter`, `rows` (one per value: parameter, `converged`, `iterations`, outputs, optional slopes `d<output>/d<input>`), `all_converged` |
| `order_study` | `rows` per polynomial degree, with corrected outputs if error estimates were requested |
| `verify` | `studies` (per-study pass/fail and wall time) and `failed`, both of this invocation; `stale_failed` (ids of failing results that earlier invocations left in the output directory) and `skipped` (studies not run, each `{id, reason}`). Pass or fail refers to this invocation only |
| `shell` | return code, duration, log tail |
| `assert` | the evaluated checks (fails the pipeline if any check fails) |
| `report` | path of the Markdown report and the collected values |

In a `sweep`, `outputs` may name the field output `f_surface` and `gradients_wrt` the shape
input `x_surface`; the rows then hold the loads and the shape derivatives
(`d<output>/dx_surface`) as lists. Slopes exist for scalar outputs only. Unknown names in
`outputs` or `gradients_wrt` are rejected before the first solve, as is `gradients_wrt` together
with outputs that contain no scalar output. `gradients_wrt` accepts `alpha_deg` (column
`d<output>/dalpha_deg`, the derivative per degree), like the `solve` stage's `gradients.wrt`.
A point that does not converge reports the outputs of its failed iterate (`null` where they are
not finite) and `null` in every derivative column; derivatives are computed for converged
points only.

<!-- end level 3 -->

<!-- begin level 3 (Mechanics) -->
**[L3 Mechanics] Records**

Each run writes `var/pipelines/<name>/pipeline.json` (status of every stage, keys, timings) and,
per stage, `stage.json`, a log (compute and `shell` stages), and whatever artefacts the stage
produces (a `solve` stage leaves the same files as `gyre run -o`). `-o` selects another
directory than `var/pipelines`.

`pipeline.json` is replaced by the next run of the pipeline; use `--record FILE` for a per-run
copy. A stage whose execution fails writes `stage.json` with `status: failed`. `var/pipelines/<name>/`
also contains a `.lock` file, which serialises the runs of the pipeline in that directory. See
[Pipelines and provenance](../engineering/pipelines-and-provenance.md) for the caching rules.

<!-- end level 3 -->