Performance
In one minuteMeasured kernel throughput on CPU and GPU, the cost of derivatives relative to a residual evaluation, and where time goes in practice — compilation for small problems, Krylov iterations for large ones.
- Small cases are compile-bound. A new combination of mesh size and polynomial degree costs tens of seconds of JIT compilation before the first iteration; the solve itself is then short. A persistent cache and warm sessions exist to pay that once.
- Derivatives cost a small multiple of a residual evaluation — the measured ratios are in the tables below.
- CPU for small problems, GPU for large ones. The service chooses per job (GPU above
60 000 degrees of freedom if one is present);
GYRE_DEVICEchooses per process. - Memory, not time, limits large cases: the Krylov basis and the element-block inverses.
- Everything below is a measurement on one host, recorded with its load; none of it is a portable performance guarantee, and no scaling study on many cores or devices exists.
The numbers on this page are measurements recorded by the verification pipeline, not portable
performance guarantees. A record written by the current timing studies states the device it
was taken on and the load of the host during the measurement (the host table), with a flag
for a host that was busy; read the timings together with that table and the run notes. Each timing is the minimum over at least 7 interleaved rounds,
and the timing studies run after all other studies of an invocation, one at a time. On a host
without a GPU the GPU study is skipped: gyre verify writes no GPU record and leaves a
performance_gpu.json of an earlier invocation in the output directory untouched. The blocks
below then show that earlier record, marked “Earlier run”, with the checks and tables it was
written with (a block for a table the record does not contain reports it as missing); with no
record in the directory they show no data.
Kernel throughput#
Wall-clock cost of the three kernels everything else is built from — the weak residual, a Jacobian-vector product (forward mode, including its primal sweep) and a vector-Jacobian product (reverse mode) — on warped periodic meshes, after compilation. Throughput is degrees of freedom per second for one residual evaluation. Each kernel is called 2 times before it is timed; the three kernels of a configuration are then timed in alternation, one call of each per round, and each timing is the minimum over the rounds: at least 7, continued until 5 consecutive rounds lower no minimum by more than 2 % (at most 60 rounds, and at most 20 s of timed calls per configuration beyond the first 7 rounds). Other work on the machine can only lengthen a call, so the minimum is the statistic least affected by it, and alternation spreads a burst of load over all three kernels. The ratio of the median to the minimum of the rounds is tabulated as a measure of the disturbance. Timings are measurements on the device and under the host load recorded with them (second table), not portable performance guarantees. Algorithmic differentiation bounds the operation count of a Jacobian-vector product by about 2.5 and of a vector-Jacobian product by about 4 to 5 residual evaluations; the two checks are regression guards about three times above those counts, leaving room for memory traffic, not derived limits.
- JVP cost / residual cost (worst case)2.392expected < 8 (regression guard, about 3 x the operation-count bound of forward mode; measured while other work loaded the machine, see the notes)pass
- VJP cost / residual cost (worst case)3.088expected < 12 (regression guard, about 3 x the operation-count bound of reverse mode; measured while other work loaded the machine, see the notes)pass
Run notes The timings were taken on a busy machine (the GPU was 17 % utilized when the study started). The reported timings are minima over interleaved rounds, which other work can raise but not lower: they are upper bounds of the undisturbed kernel costs. The median of the rounds was up to 2.20 times the minimum, which shows the size of the disturbance. The ratios of the minima, which the two checks bound from above, are less affected than the timings themselves but may still be; the checks are marked accordingly, and the throughput figures should be read as lower bounds. Timings are minima over 7 to 32 interleaved rounds.
gyre 0.4.0 · 39979fcf90 · · gpu · NVIDIA GeForce RTX 3060 Ti · quick suite · record (JSON)Same code as this build
Data table
| series | degrees of freedom | million DOF / s |
|---|---|---|
| 2-D Euler, p = 3 | 16384 | 8.19282 |
| 2-D Euler, p = 3 | 65536 | 33.3211 |
| 2-D Euler, p = 3 | 262144 | 79.9196 |
| 2-D Euler, p = 3 | 1048576 | 114.489 |
| 2-D Euler, p = 3 | 2359296 | 208.967 |
| 2-D Navier–Stokes, p = 3 | 65536 | 34.8781 |
| 2-D Navier–Stokes, p = 3 | 262144 | 73.241 |
| 2-D Navier–Stokes, p = 3 | 1048576 | 60.567 |
| 3-D Euler, p = 2 | 69120 | 20.6736 |
| 3-D Euler, p = 2 | 233280 | 51.1165 |
| 3-D Euler, p = 2 | 1866240 | 42.0342 |
gyre 0.4.0 · 39979fcf90 · · gpu · NVIDIA GeForce RTX 3060 Ti · quick suite · record (JSON)Same code as this build
| case | elements | DOF | residual [ms] | JVP [ms] | VJP [ms] | residual throughput [M DOF/s] | JVP / residual | VJP / residual | timed rounds | median / minimum (worst kernel) |
|---|---|---|---|---|---|---|---|---|---|---|
| 2-D Euler, p = 3 | 16x16 | 16384 | 1.9998 | 1.9226 | 1.8676 | 8.19282 | 0.961396 | 0.933893 | 8 | 1.07115 |
| 2-D Euler, p = 3 | 32x32 | 65536 | 1.9668 | 2.0278 | 3.0082 | 33.3211 | 1.03101 | 1.52949 | 13 | 1.55257 |
| 2-D Euler, p = 3 | 64x64 | 262144 | 3.2801 | 4.3277 | 6.9116 | 79.9196 | 1.31938 | 2.10713 | 26 | 2.02972 |
| 2-D Euler, p = 3 | 128x128 | 1048576 | 9.15879 | 21.9113 | 28.2789 | 114.489 | 2.39238 | 3.08762 | 7 | 1.41766 |
| 2-D Euler, p = 3 | 192x192 | 2359296 | 11.2903 | 21.1802 | 34.8137 | 208.967 | 1.87596 | 3.08351 | 14 | 2.07155 |
| 2-D Navier–Stokes, p = 3 | 32x32 | 65536 | 1.879 | 1.8656 | 2.7449 | 34.8781 | 0.992868 | 1.46083 | 15 | 1.86878 |
| 2-D Navier–Stokes, p = 3 | 64x64 | 262144 | 3.5792 | 5.2204 | 7.61879 | 73.241 | 1.45854 | 2.12863 | 14 | 2.19849 |
| 2-D Navier–Stokes, p = 3 | 128x128 | 1048576 | 17.3127 | 26.6045 | 45.6822 | 60.567 | 1.5367 | 2.63866 | 10 | 1.40094 |
| 3-D Euler, p = 2 | 8x8x8 | 69120 | 3.3434 | 4.1497 | 4.3832 | 20.6736 | 1.24116 | 1.311 | 8 | 1.24168 |
| 3-D Euler, p = 2 | 12x12x12 | 233280 | 4.56369 | 9.09878 | 10.1464 | 51.1165 | 1.99373 | 2.22328 | 32 | 1.82769 |
| 3-D Euler, p = 2 | 24x24x24 | 1866240 | 44.3982 | 60.4688 | 96.4992 | 42.0342 | 1.36197 | 2.1735 | 7 | 1.46526 |
gyre 0.4.0 · 39979fcf90 · · gpu · NVIDIA GeForce RTX 3060 Ti · quick suite · record (JSON)Same code as this build
| device | platform | CPUs | CPU share of other processes | load average at start (1, 5, 15 min) | load average at end | GPU utilization at start [%] | busy |
|---|---|---|---|---|---|---|---|
| NVIDIA GeForce RTX 3060 Ti | gpu | 8 | 0.0949 | 6.72 / 9.29 / 11.39 | 2.71 / 7.12 / 10.38 | 17 | yes |
gyre 0.4.0 · 39979fcf90 · · gpu · NVIDIA GeForce RTX 3060 Ti · quick suite · record (JSON)Same code as this build
Wall-clock cost of the three kernels everything else is built from — the weak residual, a Jacobian-vector product (forward mode, including its primal sweep) and a vector-Jacobian product (reverse mode) — on warped periodic meshes, after compilation. Throughput is degrees of freedom per second for one residual evaluation. Each kernel is called 2 times before it is timed; the three kernels of a configuration are then timed in alternation, one call of each per round, and each timing is the minimum over the rounds: at least 7, continued until 5 consecutive rounds lower no minimum by more than 2 % (at most 60 rounds, and at most 20 s of timed calls per configuration beyond the first 7 rounds). Other work on the machine can only lengthen a call, so the minimum is the statistic least affected by it, and alternation spreads a burst of load over all three kernels. The ratio of the median to the minimum of the rounds is tabulated as a measure of the disturbance. Timings are measurements on the device and under the host load recorded with them (second table), not portable performance guarantees. Algorithmic differentiation bounds the operation count of a Jacobian-vector product by about 2.5 and of a vector-Jacobian product by about 4 to 5 residual evaluations; the two checks are regression guards about three times above those counts, leaving room for memory traffic, not derived limits.
- JVP cost / residual cost (worst case)3.035expected < 8 (regression guard, about 3 x the operation-count bound of forward mode; measured while other work loaded the machine, see the notes)pass
- VJP cost / residual cost (worst case)7.329expected < 12 (regression guard, about 3 x the operation-count bound of reverse mode; measured while other work loaded the machine, see the notes)pass
Run notes The timings were taken on a busy machine (other processes used 12 % of the capacity of the 8 CPUs while the study ran). The reported timings are minima over interleaved rounds, which other work can raise but not lower: they are upper bounds of the undisturbed kernel costs. The median of the rounds was up to 1.85 times the minimum, which shows the size of the disturbance. The ratios of the minima, which the two checks bound from above, are less affected than the timings themselves but may still be; the checks are marked accordingly, and the throughput figures should be read as lower bounds. Timings are minima over 11 to 20 interleaved rounds.
gyre 0.4.0 · 39979fcf90 · · cpu · quick suite · record (JSON)Same code as this build
Data table
| series | degrees of freedom | million DOF / s |
|---|---|---|
| 2-D Euler, p = 3 | 16384 | 4.62864 |
| 2-D Euler, p = 3 | 65536 | 6.77207 |
| 2-D Euler, p = 3 | 262144 | 2.38102 |
| 2-D Navier–Stokes, p = 3 | 65536 | 2.87552 |
| 2-D Navier–Stokes, p = 3 | 262144 | 1.15204 |
| 3-D Euler, p = 2 | 69120 | 3.02713 |
| 3-D Euler, p = 2 | 233280 | 1.57031 |
gyre 0.4.0 · 39979fcf90 · · cpu · quick suite · record (JSON)Same code as this build
| case | elements | DOF | residual [ms] | JVP [ms] | VJP [ms] | residual throughput [M DOF/s] | JVP / residual | VJP / residual | timed rounds | median / minimum (worst kernel) |
|---|---|---|---|---|---|---|---|---|---|---|
| 2-D Euler, p = 3 | 16x16 | 16384 | 3.5397 | 6.4003 | 10.6923 | 4.62864 | 1.80815 | 3.02068 | 11 | 1.53423 |
| 2-D Euler, p = 3 | 32x32 | 65536 | 9.6774 | 29.3689 | 70.9214 | 6.77207 | 3.03479 | 7.32856 | 11 | 1.85188 |
| 2-D Euler, p = 3 | 64x64 | 262144 | 110.097 | 188.667 | 420.119 | 2.38102 | 1.71364 | 3.81589 | 11 | 1.11999 |
| 2-D Navier–Stokes, p = 3 | 32x32 | 65536 | 22.791 | 62.0382 | 90.5424 | 2.87552 | 2.72205 | 3.97273 | 13 | 1.60486 |
| 2-D Navier–Stokes, p = 3 | 64x64 | 262144 | 227.548 | 355.136 | 591.178 | 1.15204 | 1.56071 | 2.59804 | 12 | 1.28127 |
| 3-D Euler, p = 2 | 8x8x8 | 69120 | 22.8335 | 49.8549 | 98.9936 | 3.02713 | 2.18341 | 4.33546 | 20 | 1.48412 |
| 3-D Euler, p = 2 | 12x12x12 | 233280 | 148.557 | 258.628 | 353.32 | 1.57031 | 1.74094 | 2.37834 | 15 | 1.30874 |
gyre 0.4.0 · 39979fcf90 · · cpu · quick suite · record (JSON)Same code as this build
| device | platform | CPUs | CPU share of other processes | load average at start (1, 5, 15 min) | load average at end | GPU utilization at start [%] | busy |
|---|---|---|---|---|---|---|---|
| cpu | cpu | 8 | 0.1184 | 9.36 / 11.72 / 12.43 | 7.04 / 9.44 / 11.46 | — | yes |
gyre 0.4.0 · 39979fcf90 · · cpu · quick suite · record (JSON)Same code as this build
A Jacobian–vector product (forward mode, including the primal sweep it needs) and a
vector–Jacobian product (reverse mode) each cost a small multiple of one residual evaluation —
the columns JVP / residual and VJP / residual of the tables. The two checks of each study
bound those ratios as regression guards, about three times above the operation counts of
forward and reverse mode; they are not derived limits. Inside the Krylov solver the primal sweep is shared by all
iterations (linearise inside the solve), so the per-iteration
cost is lower than the stand-alone JVP measured here.
| Problem size | Dominant cost | What helps |
|---|---|---|
| small (up to ~10⁴–10⁵ DOF) | JIT compilation, tens of seconds per new mesh/order | persistent cache, sessions, CPU execution |
| medium | Krylov iterations × tangent sweeps; block-inverse set-up | lagging the block inverses (solver.precond_lag), GPU; on wall-resolved meshes the line smoother for solves to a tight tolerance (measured on two small cases); p-multigrid lowers the Krylov iteration count; its net effect on operator applications and on compiled wall time is measured per case, not claimed in general |
| large | memory: Krylov basis and block inverses ( block size²) | solver.precond_precision: single halves the blocks and the working memory of their assembly without changing the answers (comparison, largest solve); lower krylov_dim; more device memory |
The service picks the device per job from the estimated number of degrees of freedom (GPU above 60 000 if one is present).
The element blocks of the preconditioner come from batched Jacobian-vector products: one tangent per column of a block, for all elements at once. One tangent of the flux-differencing residual holds two-point fluxes per node and direction, so the working memory of a batch grows with
and in three dimensions at higher degree a batch of all columns no longer fits: at on
135 000 unknowns the assembly asked an 8 GB device for 8.1 GiB. The batch is therefore sized
to keep that product below a budget (gyre.implicit.BLOCK_BATCH_BUDGET, ;
about 1 GiB for the largest single allocation, by measurement on that device), and the
columns are processed in several sweeps. The blocks do not depend on the batch; only memory
and time do. GYRE_BLOCK_BATCH_BUDGET raises the budget on a device with more memory. The
blocks themselves, numbers and as many again for their inverses,
remain the floor: 675 MB each at for 216 hexahedra.
Every kernel is a sequence of element-wise operations and small tensor contractions over arrays with a leading element axis — the pattern accelerators execute best. There is no sparse matrix assembly, no indirect addressing beyond one gather for neighbour traces, and no per-element Python control flow. The same source runs on both devices; the choice is an environment variable.