--- title: Design decisions summary: The handful of decisions that shape everything else — JAX as the kernel language, split-form DGSEM, geometry as differentiable data, split-argument residuals, one execution path, isolated worker processes — with the alternatives that were rejected. section: engineering order: 1 level: 2 keywords: [architecture decision records, JAX, design, trade-offs] --- Each decision below records the context, the choice and what it costs. They are kept current in `devlog/ROADMAP.md`. | # | Decision | Main consequence | |---|---|---| | 1 | JAX (double precision) as the kernel language | exact JVP/VJP of every operator; CPU and GPU from one source | | 2 | Split-form DGSEM on Gauss–Lobatto nodes | entropy-conservative volume terms (a proven entropy inequality with the `es_roe` and `llf` interface fluxes, not with the default `roe`); tensor-product, dimension-generic kernels | | 3 | BR2 with strong gradient and weak divergence | compact stencil, symmetric (adjoint-consistent) diffusion | | 4 | Split-argument residual `R(U_own, U_nbr)` | exact element blocks from plain JVPs; no colouring, no assembly | | 5 | Geometry enters as data inside the traced function | shape derivatives without dedicated code | | 6 | Weak-form residual for all linear algebra | balanced norms, Galerkin restriction is $P^{\mathsf T}$, adjoint is a nodal field | | 7 | One execution path (`CaseSpec` → `Case` → `run_case`) | CLI, service, pipelines and agents cannot diverge | | 8 | Solver work only in subprocesses | isolation, per-job device choice, trivial cancellation | | 9 | Evidence generated by pipeline, rendered by the docs | published numbers always carry provenance | | 10 | The turbulence closure is one more row of the same residual | Newton, adjoint and shape derivatives stay exact with the model on, wall distance included; no second code path | | 11 | The classical Roe flux stays the default; the entropy-stable `es_roe` is an option | the default is not provably entropy stable; switching it is a measured trade, not a free improvement | | 12 | A linear solve that misses its tolerance is an error | derivatives are never returned from a stalled solve; weak preconditioning shows up as a failure instead of a wrong number | | 14 | The preconditioner may be single precision; everything it preconditions is double | the blocks are what limits 3-D size on one device and the only part that need not be exact; the default stays double until the option has been measured more widely | | 13 | Line smoothing is built from the mesh and from AD, and is an option | wall-resolved meshes get an exact solve along the wall-normal lines without an assembled matrix; the default stays block Jacobi because the gain was measured on two small cases and only for tight linear solves | **[L3 Mechanics] 1 — JAX as the kernel language** **Context.** An MDO discipline needs Jacobian-vector products for Newton–Krylov, transposed products for adjoints, and derivatives with respect to geometry and flow parameters. Hand-differentiated solvers carry a permanent maintenance burden and a permanent risk that the derivative code lags the primal code. **Decision.** Write every numerical kernel as a pure function of JAX arrays; obtain all derivatives with `jax.jvp`, `jax.vjp`, `jax.linearize` and `jax.grad`; vectorise with array programming and `vmap`. **Cost.** Just-in-time compilation: tens of seconds for each new combination of mesh size and polynomial degree, which dominates small runs. Mitigations: a persistent compilation cache, warm sessions, CPU execution for small problems. Static shapes rule out some adaptive algorithms (e.g. per-element polynomial degree) without restructuring. **Rejected.** A C++ core with source-transformation or operator-overloading AD: faster cold start, but far more code, a separate adjoint build, and no free GPU path. **[L3 Mechanics] 2 — Split-form DGSEM** **Context.** High-order methods are attractive for design (accuracy per degree of freedom, smooth dependence of outputs on inputs) but classical collocation schemes are fragile when under-resolved. **Decision.** Flux-differencing DGSEM on Gauss–Lobatto nodes with entropy-conservative two-point fluxes in the volume and upwind dissipation at interfaces. **Cost.** Two-point fluxes are evaluated for node pairs rather than nodes (mitigated by the symmetric pair tables), and Gauss–Lobatto collocation is somewhat less accurate per degree of freedom than Gauss collocation. **Rejected.** Standard (over-integrated) DG: robust only with expensive over-integration and no discrete entropy statement. Finite volume: far simpler, but low order. **[L3 Mechanics] 4 — Split-argument residual** **Context.** Matrix-free Krylov solvers need a preconditioner; the workhorse for DG is element block Jacobi, usually inside multigrid. Extracting blocks from an AD Jacobian normally needs graph colouring and many JVPs, or sparse assembly. **Decision.** Give the residual two state arguments: one used for everything element-local, one for everything gathered from neighbours. The partial derivative with respect to the first is exactly block diagonal, so `block_size` JVPs with identical per-element unit tangents return all blocks. **Cost.** Every boundary condition and every face term must respect the convention (covered by a test that compares against a dense Jacobian for each wall type), and each interior face is evaluated from both sides. **[L3 Mechanics] 7 and 8 — One path, isolated processes** **Context.** A capability exposed through four interfaces tends to grow four subtly different behaviours; and a numerical solver inside a web server can stall or crash it. **Decision.** All interfaces construct a `CaseSpec` and call the same runner. The API process never executes solver kernels: jobs run in fresh worker processes, sessions in long-lived ones, each with its own JAX runtime and device selection. MCP tools call the REST API through an in-process ASGI transport, so agents exercise the same code path as programs. **Cost.** Process start-up and JIT per job (hence sessions and the result cache), and data crossing process boundaries as JSON. **[L3 Mechanics] 10 — The turbulence closure as a row of the residual** **Context.** A RANS model brings stiff sources, a wall distance and a variable six orders smaller than the mean flow. Solvers often treat it apart: loosely coupled, frozen in the adjoint, with a wall distance that is not differentiated. **Decision.** The model variable is appended to the state and its equation to the residual ([formulation](../theory/turbulence-model.md)). Sources are integrated at Gauss points with the BR2 gradient; the wall distance is stored as foot-point weights on mesh nodes and re-evaluated inside the differentiable geometry; solver norms weight the rows. **Cost.** Element blocks one variable larger, sources evaluated at a second set of points, and a residual whose switches had to be smoothed where the model has kinks. One model only. **[L3 Mechanics] 11 — Which interface flux is the default** **Context.** With an entropy-conservative volume flux the scheme is provably entropy stable only if the interface flux is. The classical Roe flux is not; `es_roe`, added later, is, and has the same wave-by-wave dissipation. **Decision.** Keep `roe` as the default and offer `es_roe`. Measured on the development machine before deciding: on four subsonic presets the two fluxes needed the same Newton steps and Krylov iterations to within 4 % and gave outputs within the discretization error; the transonic preset, which converges with `roe` in 238 of its 250 steps, did not converge within 250 with `es_roe`; the residual kernel of `es_roe` took 1.04 times as long as that of `roe` on a 2-D case at $p = 3$ and 1.67 times on a 3-D case at $p = 2$, where faces carry more of the work (working tree before revision `f36e10e`, RTX 3060 Ti). **Cost.** The default configuration carries no entropy proof, which every page that mentions stability has to say. Revisit when the transonic case converges with the entropy-stable flux. **[L3 Mechanics] 12 — A stalled linear solve is an error** **Context.** Total derivatives are exact up to the tolerance of a linear solve. A solve that stops short returns numbers that look like derivatives. **Decision.** `solve_linear`, the total products and `totals` raise `LinearSolveError` with the reduction they achieved; returning the result with a flag is something the caller asks for. The OpenMDAO components turn it into an analysis error. **Cost.** Cases that used to return a number now stop. The first was the adjoint of the turbulent flat plate with block Jacobi alone ([what it showed](../verification/turbulence.md#adjoint-derivatives)). **[L3 Mechanics] 13 — Line smoothing as an option** **Context.** On the wall-resolved mesh of the turbulent flat plate, block Jacobi alone left the adjoint solve short of its tolerance (decision 12 made that visible), and the piecewise-constant multigrid level only repaired it. The standard remedy for such meshes is to solve implicitly along lines of strongly coupled elements. Gyre assembles no matrix, so the usual implementation, which picks the line blocks out of a stored Jacobian, was not available. **Decision.** Build the lines from the geometry of the mesh (face area over centre distance, a threshold on the anisotropy, links only where both elements agree), take the two off-diagonal blocks of a line element from derivatives of the split-argument residual with respect to the neighbour state, with one batch of tangents per colour of a colouring of the line elements, and invert the block-tridiagonal systems exactly. The lines do not depend on the flow, the same code serves every multigrid level and the transposed solve, and a mesh without anisotropy gets block Jacobi back unchanged. The option is `solver.smoother: line`; the default smoother and the preset `flat_plate_sa` are unchanged. **Cost.** Three more batches of block derivatives at every update of the preconditioner (three colours on aligned lines), two more blocks of storage per stored line slot ([how the slots are bounded](../theory/implicit-solver.md#preconditioning-without-assembling-a-matrix)), and a longer first compilation. Two comparisons are recorded, the turbulent flat plate in two dimensions and a laminar sphere in three. In both, the linear solves to a tight tolerance need fewer than half the Krylov iterations ([study](../verification/solvers.md#line-smoothing-on-a-wall-resolved-mesh)), and in both the flow solve from the free stream, whose loose Newton steps spend their time in assembling the preconditioner, gains nothing: **Table: Flow solve from the free stream** (Line-implicit smoothing on a wall-resolved mesh) | preconditioner | nonlinear steps | Krylov iterations | converged | drag coefficient | wall time, first solve [s] | wall time, repeated solve (compiled) [s] | repeated solve converged | |---|---|---|---|---|---|---|---| | block Jacobi, levels 2-1-0 | 34 | 540 | yes | 0.00575365 | 86.6006 | 33.1422 | yes | | line Jacobi, levels 2-1-0 | 34 | 267 | yes | 0.00575365 | 159.257 | 233.618 | yes | | block Jacobi, one level | 40 | 3498 | yes | 0.00575365 | 136.85 | 113.283 | yes | | line Jacobi, one level | 34 | 1914 | yes | 0.00575365 | 245.519 | 392.492 | yes | | line Jacobi, levels 2-1-0, blocks every 2nd step | 34 | 268 | yes | 0.00575365 | 189.424 | 147.94 | yes | Provenance: gyre 0.4.0; revision `39979fcf90` (commit `39979fcf902ad6116eb550b881b4278d4599c8e2`); 2026-11-12T10:58:43-0500; device cpu; quick suite. **Revision check: Same code as this build.** Produced by 39979fcf902ad6116eb550b881b4278d4599c8e2. Every commit between it and the commit this site was built from (ceb1f4b0bed41de53e586df10858079f5a2bcdd8) changes only documentation text, evidence records or bookkeeping: the code is the same. Full record: [line_smoother.json](../data/verification/line_smoother.json). **Table: Flow solve from the free stream** (Line-implicit smoothing in three dimensions) | preconditioner | levels | nonlinear steps | Krylov iterations | converged | drag coefficient | wall time, first solve [s] | wall time, repeated solve (compiled) [s] | repeated solve converged | |---|---|---|---|---|---|---|---|---| | block Jacobi | 2, 1, 0 | 13 | 193 | yes | 1.13546 | 141.43 | 13.8529 | yes | | line Jacobi | 2, 1, 0 | 13 | 123 | yes | 1.13546 | 109.475 | 19.5358 | yes | Provenance: gyre 0.4.0; revision `39979fcf90` (commit `39979fcf902ad6116eb550b881b4278d4599c8e2`); 2026-11-12T10:58:47-0500; device gpu · NVIDIA GeForce RTX 3060 Ti; quick suite. **Revision check: Same code as this build.** Produced by 39979fcf902ad6116eb550b881b4278d4599c8e2. Every commit between it and the commit this site was built from (ceb1f4b0bed41de53e586df10858079f5a2bcdd8) changes only documentation text, evidence records or bookkeeping: the code is the same. Full record: [line_smoother_3d.json](../data/verification/line_smoother_3d.json). Making it the default for wall-resolved cases needs the same measurement on a large mesh and on a turbulent case in three dimensions, which has not been made. **[L3 Mechanics] 14 — Single precision where exactness is not needed** **Context.** In three dimensions the element blocks of the smoother, and the working memory of their assembly, decide what fits on one device. Everything else the solver keeps is a few dozen state vectors. **Decision.** `solver.precond_precision: single` rounds the blocks, column by column as they are assembled, stores them and applies them in single precision. The inverses and the factorisation of the line systems are still computed in double precision, because that is where a badly conditioned block loses digits, and the operator, the Krylov vectors and the transfers are never touched: a rounded preconditioner changes how fast flexible GMRES converges, not what it converges to. The option is part of the case, so a session, the error estimate and the adjoint use it consistently. **Cost and evidence.** On the three cases of the [comparison](../verification/solvers.md#single-precision-blocks) the iteration counts and the answers do not change. The [largest 3-D solve](../verification/three-dimensional.md#largest-solve-on-one-device) is run with it. The default remains `double`: three small cases and one large one are not every flow, shocks with artificial viscosity among them, and a default that changes the iteration path of every existing case should be changed on more evidence than that. **[L2 Practice] What was deliberately left out** No optimizer, no design-variable abstraction, no geometry engine: those belong to the MDO framework. No bespoke workflow scheduler beyond the small pipeline engine. No database: the store is files.