Troubleshooting
In one minuteWhat to do when a solve does not converge — every reason the solver can report, what it means and which control to change — and the first-run and service symptoms most often met.
A solve that does not converge is reported, not raised: the result carries converged: false
and a reason. gyre run exits with status 2; its printed summary shows converged: false
but not the reason, which is in result.json (run with -o DIR). A job result, a session
evaluation and SolveHistory.reason in Python carry the same text.
The reasons a solve can end with#
reason |
Converged? | Meaning |
|---|---|---|
residual tolerance reached |
yes | the residual fell below tol_rel × the reference residual |
residual stagnated at round-off floor |
yes | the residual stopped halving, at or below of the reference residual |
maximum number of steps reached |
no | max_steps nonlinear steps were used |
residual stalled above tolerance |
no | the residual stopped halving above of the reference residual |
residual is not finite |
no | the residual of the initial state is nan or inf |
pseudo-time step collapsed (no admissible update) |
no | no step was accepted even at a pseudo-time CFL below |
These six strings are the complete list (gyre.implicit.SteadySolver.solve).
The controls are the fields of solver and discretization in the case; on the
command line --set solver.max_steps=400. The remedies are listed in the order to try them.
They follow from how the iteration works (see the block below); they are not the outcome of a
study of failures.
maximum number of steps reached. Look at the residual history first (gyre run prints
it; result.json and the job events hold it).
- The residual is still falling and the CFL still growing: the solve needed more steps. Raise
solver.max_steps(default 150). - The CFL stays small or keeps being cut: the start is too far from the solution. Start from
a converged neighbour instead (a session or
flow.solve()afterset_inputswarm-starts from the last converged state), approach the target condition in smaller increments, or lowersolver.cfl0(default 5). - The Krylov iteration count per step sits at its limit (
krylov_dim×restarts, default 40 × 2) and the residual falls slowly: the linear systems are not being solved. Raisesolver.krylov_dim, or usesolver.smoother: gmres; keepsolver.levels: auto. On a wall-resolved mesh (cells much longer than thick at the wall) usesolver.smoother: line, which solves exactly along the lines of thin cells (measured on the turbulent flat plate).
residual stalled above tolerance. The residual did not halve in six consecutive steps
although the CFL had reached . Usual causes: an unsteady or marginally stable flow that
has no steady state at this resolution (a wake that wants to shed, a shock that moves between
nodes), or linear solves that do not reach lin_tol.
- Check the last entries of the history:
linear_reductionwell abovelin_tolmeans the linear solver is the limit — raisesolver.krylov_dimorsolver.restarts. - A flow with a shock needs
discretization.shock_capturing: true; a largerdiscretization.av_coef(default 1) smooths the shock further at the price of accuracy. - A coarser tolerance is a decision about the result, not a fix: with
tol_relabove the plateau the solve “converges” to the same state. Compare the plateau with the accuracy you need before doing that.
residual is not finite. This reason describes the starting point, not the
iteration: the residual of the initial state — the uniform free stream, or the warm-start
state evaluated with the new inputs — is nan or inf. The line search never accepts an
update with a non-finite residual, so the reason cannot appear after the first step, and
no solver control (cfl0, cfl_growth, max_steps) changes it.
- Check the inputs: the Mach number,
gamma, and the boundary data inflow.extramust give positive density and pressure. - Check the mesh:
gyre mesh inforeports inverted or badly scaled elements (messages and remedies). - Check the start state: a state restored from a checkpoint or set by hand, or the iterate of a failed solve passed on as a start, may itself be inadmissible.
pseudo-time step collapsed (no admissible update). Every update, at every step length,
either produced negative density or pressure, gave a non-finite residual, or failed to reduce
the residual, and cutting the pseudo-time step did not help. This is how a transient that
leaves the admissible range ends (the scheme does not enforce positivity); if the step limit
is reached first, the same cause is reported as maximum number of steps reached.
- Start from a converged neighbouring condition, or lower
solver.cfl0andsolver.cfl_growth(default 10) so that the transient is followed more closely. - Under-resolved strong gradients: enable
discretization.shock_capturing, lower the polynomial degree (discretization.order), or refine the mesh where the gradient is. - In a design loop it usually means the last step in the design variables was too large — the kept state is still the last converged one, so take a smaller step.
Converged, but residual stagnated at round-off floor. Not a failure: the residual
reached the conditioning-limited floor before tol_rel. The result is as converged as double
precision allows on this mesh; a smaller tol_rel cannot be met.
- The outputs in the result are those of the iterate the failed attempt ended at; they are not a solution.
- The discipline keeps the last converged state and warm-starts the next solve from it. Repeating the same request repeats the same failure.
- Gradients and error estimates are not computed for a failed point: the result record
(
result.json, a job result) and the summary printed bygyre runlist them underskipped, next to thereasonthe solve ended with; sessions answer 422, and the Python calls raiseRuntimeError(Python API, MDO integration). - Order selection (
--target-error) ends at the first degree that does not converge and reports the last degree that did.
Each step solves a linearised pseudo-time problem to the relative tolerance lin_tol and
tries the update with step lengths 1, ½, …, down to 1/32. A step length is accepted if the
new state has positive density and pressure, a finite residual, and a smaller pseudo-transient
residual. After an accepted full step the CFL grows by the factor by which the residual fell,
limited to between 1.5 and cfl_growth; after a shortened step it stays (a step shortened to a quarter or less multiplies it by
solver.cfl_shrink, which is 1 by default); if no step length
is accepted the CFL is divided by ten and the step is repeated. Stagnation is declared after
six steps without halving the best residual once the CFL is at least . The reference
residual is that of the uniform free stream, also for warm starts.
See the implicit solver.
First run and installation#
| Symptom | Cause | Remedy |
|---|---|---|
import jax aborts with “illegal instruction” |
the CPU has no AVX2 and a newer JAX wheel was installed | make bootstrap detects this and pins jax < 0.12 (constraints/avx-only.txt); with pip, install with -c constraints/avx-only.txt |
gyre: command not found |
the virtual environment is not activated | call .venv/bin/gyre, or source .venv/bin/activate once per shell |
nothing is printed for tens of seconds after the case … line |
kernels are being compiled; the CLI prints nothing while JAX compiles | wait: see the measured times in Getting started. Later runs of the same mesh size and degree reuse the cache in ~/.cache/gyre/jit |
| a small case is slower on the GPU than on the CPU | compilation dominates and is slower for the GPU | GYRE_DEVICE=cpu (set before the process starts; the choice is per process) |
| a 3-D solve exhausts device memory while the preconditioner is assembled | the element blocks of the smoother grow like per element | set solver.precond_precision: single (half the memory, same answers); lower the degree or the number of elements |
| a derivative call exhausts memory | forward mode over many inputs (jax.jacfwd with respect to all mesh nodes) builds one tangent per input |
use reverse mode (jax.grad, flow.totals, adjoint_product) for one output and many inputs (Python API) |
| stale or corrupt compiled kernels are suspected | the persistent cache | GYRE_JIT_CACHE=off disables it for a process; GYRE_JIT_CACHE=<dir> uses another directory |
Service symptoms#
| Answer | Meaning | What to do |
|---|---|---|
429 QueueFull |
the job queue is at its limit | wait for jobs to finish and submit again, or raise GYRE_MAX_QUEUED_JOBS |
429 TooManySessions |
the session limit is reached | close sessions that are no longer used (DELETE /api/v1/sessions/{id}); the operator can raise GYRE_MAX_SESSIONS |
504 Timeout |
a session command, or the start-up of a session, exceeded its time limit | check the size of the case with POST /api/v1/cases/validate; the first evaluation of a new size compiles kernels |
410 SessionDead |
the solver process of the session has ended and its state with it | open a new session |
409 NotReady |
the job has not finished, or did not succeed | poll GET /api/v1/jobs/{id}; read error in the job record if its state is failed |
422 with type: RuntimeError on a session command |
derivatives or an error estimate were requested after an evaluation that did not converge | evaluate a point that converges first |
The complete list of error types is in the service guide; the limits and their environment variables are in the deployment guide.