Gyredocs
Depth Practice
API

Gyre/Verification

Solver convergence

In one minuteThe pseudo-transient Newton–Krylov solver reduces the residual by ten orders from a free stream; its terminal convergence is linear at the forcing rate of the inexact Newton iteration, and p-multigrid preconditioning cuts the Krylov iteration count compared with block Jacobi alone. What those iterations cost is measured and reported without a threshold: operator applications counted from the cycle, and the wall time of each solve repeated with compiled kernels.

Written at L2 Practice goes to L3 Mechanics 4791 words Markdown
PassedNonlinear and linear solver convergence16 of 16 checks passed · 890.1 s · quick suite

Residual histories of the pseudo-transient Newton–Krylov solver from a uniform free stream, with element-block-Jacobi and with p-multigrid preconditioned FGMRES. The linear systems are solved to a fixed relative tolerance (inexact Newton), so once the pseudo-time step is large the residual is expected to fall by about that factor per step: linear convergence at the forcing rate, not quadratic convergence. The checks require a residual reduction of ten orders, a terminal contraction per step within a factor five of the forcing term (the signature of an exact Jacobian), and fewer than half the Krylov iterations with p-multigrid. What those iterations cost is measured and reported without a threshold: operator applications counted from the definition of the multigrid cycle, and the wall time of each solve repeated with compiled kernels.

  • Euler NACA 0012, p = 2, block Jacobi: converged 10 orders (final / reference residual)2.86e-12expected <= 1e-10pass
  • Euler NACA 0012, p = 2, block Jacobi: terminal residual contraction per step (geometric mean)0.01092expected <= 0.05 (5 x forcing term lin_tol = 0.01)pass
  • Euler NACA 0012, p = 2, p-multigrid: converged 10 orders (final / reference residual)1.93e-12expected <= 1e-10pass
  • Euler NACA 0012, p = 2, p-multigrid: terminal residual contraction per step (geometric mean)0.0102expected <= 0.05 (5 x forcing term lin_tol = 0.01)pass
  • Euler NACA 0012, p = 2: Krylov iterations, p-MG / block Jacobi0.1177expected < 0.5pass
  • Laminar cylinder Re = 40, p = 2, block Jacobi: converged 10 orders (final / reference residual)1.52e-12expected <= 1e-10pass
  • Laminar cylinder Re = 40, p = 2, block Jacobi: terminal residual contraction per step (geometric mean)0.009805expected <= 0.05 (5 x forcing term lin_tol = 0.01)pass
  • Laminar cylinder Re = 40, p = 2, p-multigrid: converged 10 orders (final / reference residual)8.21e-11expected <= 1e-10pass
  • Laminar cylinder Re = 40, p = 2, p-multigrid: terminal residual contraction per step (geometric mean)0.007633expected <= 0.05 (5 x forcing term lin_tol = 0.01)pass
  • Laminar cylinder Re = 40, p = 2: Krylov iterations, p-MG / block Jacobi0.08377expected < 0.5pass
  • Euler NACA 0012, p = 3, block Jacobi: converged 10 orders (final / reference residual)3.30e-12expected <= 1e-10pass
  • Euler NACA 0012, p = 3, block Jacobi: terminal residual contraction per step (geometric mean)0.01179expected <= 0.05 (5 x forcing term lin_tol = 0.01)pass
  • Euler NACA 0012, p = 3, p-multigrid: converged 10 orders (final / reference residual)5.90e-11expected <= 1e-10pass
  • Euler NACA 0012, p = 3, p-multigrid: terminal residual contraction per step (geometric mean)0.005823expected <= 0.05 (5 x forcing term lin_tol = 0.01)pass
  • Euler NACA 0012, p = 3: Krylov iterations, p-MG / block Jacobi0.1132expected < 0.5pass
  • cost of the two preconditioners measured: the repeated, compiled solves convergedyesexpected truepass

Run notes Measured in this run: below 0.0001 of the reference residual the residual fell by a factor 5.8e-03 to 1.2e-02 per step (geometric means; worst single step 1.7e-02) with a forcing term of 0.01. p-multigrid needed 0.08 to 0.12 of the Krylov iterations of block Jacobi. That is a ratio of iteration counts, not of work: one multigrid-preconditioned iteration applies the operator several times (smoothing sweeps on every level). In fine-level operator applications the ratio was 0.63 to 0.89, and in wall time of the repeated solve, with every kernel compiled, 0.88 to 1.38 (below 1: p-multigrid cheaper). The wall times of the first solves in the effort table include one-off JIT compilation. Machine load while the study ran: The timings were taken on a busy machine (other processes used 51 % of the capacity of the 8 CPUs while the study ran).

gyre 0.4.0 · 39979fcf90 · · cpu · quick suite · record (JSON)Same code as this build

Residual historyNonlinear and linear solver convergence
Data table
seriesnonlinear iteration|Rw|
Euler NACA 0012, p = 2: block Jacobi00.366317
Euler NACA 0012, p = 2: block Jacobi10.238266
Euler NACA 0012, p = 2: block Jacobi20.153701
Euler NACA 0012, p = 2: block Jacobi30.128176
Euler NACA 0012, p = 2: block Jacobi40.107526
Euler NACA 0012, p = 2: block Jacobi50.0911261
Euler NACA 0012, p = 2: block Jacobi60.0708977
Euler NACA 0012, p = 2: block Jacobi70.0507337
Euler NACA 0012, p = 2: block Jacobi80.0305722
Euler NACA 0012, p = 2: block Jacobi90.0140312
Euler NACA 0012, p = 2: block Jacobi100.00539924
Euler NACA 0012, p = 2: block Jacobi119.9688e-04
Euler NACA 0012, p = 2: block Jacobi126.0154e-05
Euler NACA 0012, p = 2: block Jacobi138.0257e-07
Euler NACA 0012, p = 2: block Jacobi147.8549e-09
Euler NACA 0012, p = 2: block Jacobi157.7948e-11
Euler NACA 0012, p = 2: block Jacobi161.0461e-12
Euler NACA 0012, p = 2: p-multigrid00.366317
Euler NACA 0012, p = 2: p-multigrid10.238446
Euler NACA 0012, p = 2: p-multigrid20.15374
Euler NACA 0012, p = 2: p-multigrid30.127815
Euler NACA 0012, p = 2: p-multigrid40.10725
Euler NACA 0012, p = 2: p-multigrid50.0910524
Euler NACA 0012, p = 2: p-multigrid60.0707134
Euler NACA 0012, p = 2: p-multigrid70.0507297
Euler NACA 0012, p = 2: p-multigrid80.0306123
Euler NACA 0012, p = 2: p-multigrid90.0141012
Euler NACA 0012, p = 2: p-multigrid100.00544363
Euler NACA 0012, p = 2: p-multigrid110.00100033
Euler NACA 0012, p = 2: p-multigrid126.0277e-05
Euler NACA 0012, p = 2: p-multigrid136.6459e-07
Euler NACA 0012, p = 2: p-multigrid146.4517e-09
Euler NACA 0012, p = 2: p-multigrid154.5689e-11
Euler NACA 0012, p = 2: p-multigrid167.0588e-13
Laminar cylinder Re = 40, p = 2: block Jacobi020.2116
Laminar cylinder Re = 40, p = 2: block Jacobi113.1423
Laminar cylinder Re = 40, p = 2: block Jacobi29.47857
Laminar cylinder Re = 40, p = 2: block Jacobi36.10869
Laminar cylinder Re = 40, p = 2: block Jacobi44.09279
Laminar cylinder Re = 40, p = 2: block Jacobi53.38947
Laminar cylinder Re = 40, p = 2: block Jacobi61.97246
Laminar cylinder Re = 40, p = 2: block Jacobi70.483745
Laminar cylinder Re = 40, p = 2: block Jacobi80.141892
Laminar cylinder Re = 40, p = 2: block Jacobi90.074575
Laminar cylinder Re = 40, p = 2: block Jacobi100.0199166
Laminar cylinder Re = 40, p = 2: block Jacobi110.0024487
Laminar cylinder Re = 40, p = 2: block Jacobi123.2684e-05
Laminar cylinder Re = 40, p = 2: block Jacobi133.2557e-07
Laminar cylinder Re = 40, p = 2: block Jacobi143.2121e-09
Laminar cylinder Re = 40, p = 2: block Jacobi153.0809e-11
Laminar cylinder Re = 40, p = 2: p-multigrid020.2116
Laminar cylinder Re = 40, p = 2: p-multigrid113.1323
Laminar cylinder Re = 40, p = 2: p-multigrid29.44805
Laminar cylinder Re = 40, p = 2: p-multigrid36.06778
Laminar cylinder Re = 40, p = 2: p-multigrid44.06751
Laminar cylinder Re = 40, p = 2: p-multigrid53.34489
Laminar cylinder Re = 40, p = 2: p-multigrid61.94072
Laminar cylinder Re = 40, p = 2: p-multigrid70.477141
Laminar cylinder Re = 40, p = 2: p-multigrid80.141383
Laminar cylinder Re = 40, p = 2: p-multigrid90.0735677
Laminar cylinder Re = 40, p = 2: p-multigrid100.0196538
Laminar cylinder Re = 40, p = 2: p-multigrid110.00245037
Laminar cylinder Re = 40, p = 2: p-multigrid122.8469e-05
Laminar cylinder Re = 40, p = 2: p-multigrid131.9200e-07
Laminar cylinder Re = 40, p = 2: p-multigrid141.6586e-09
Euler NACA 0012, p = 3: block Jacobi00.311716
Euler NACA 0012, p = 3: block Jacobi10.313325
Euler NACA 0012, p = 3: block Jacobi20.138704
Euler NACA 0012, p = 3: block Jacobi30.0970839
Euler NACA 0012, p = 3: block Jacobi40.074958
Euler NACA 0012, p = 3: block Jacobi50.0617388
Euler NACA 0012, p = 3: block Jacobi60.0472716
Euler NACA 0012, p = 3: block Jacobi70.0336438
Euler NACA 0012, p = 3: block Jacobi80.0197131
Euler NACA 0012, p = 3: block Jacobi90.0086172
Euler NACA 0012, p = 3: block Jacobi100.00352864
Euler NACA 0012, p = 3: block Jacobi116.8180e-04
Euler NACA 0012, p = 3: block Jacobi124.6864e-05
Euler NACA 0012, p = 3: block Jacobi136.2728e-07
Euler NACA 0012, p = 3: block Jacobi146.1512e-09
Euler NACA 0012, p = 3: block Jacobi156.0011e-11
Euler NACA 0012, p = 3: block Jacobi161.0292e-12
Euler NACA 0012, p = 3: p-multigrid00.311716
Euler NACA 0012, p = 3: p-multigrid10.313727
Euler NACA 0012, p = 3: p-multigrid20.13878
Euler NACA 0012, p = 3: p-multigrid30.0967875
Euler NACA 0012, p = 3: p-multigrid40.074779
Euler NACA 0012, p = 3: p-multigrid50.0616541
Euler NACA 0012, p = 3: p-multigrid60.0471596
Euler NACA 0012, p = 3: p-multigrid70.0336347
Euler NACA 0012, p = 3: p-multigrid80.0197123
Euler NACA 0012, p = 3: p-multigrid90.00862647
Euler NACA 0012, p = 3: p-multigrid100.00352607
Euler NACA 0012, p = 3: p-multigrid116.7471e-04
Euler NACA 0012, p = 3: p-multigrid124.6216e-05
Euler NACA 0012, p = 3: p-multigrid135.4278e-07
Euler NACA 0012, p = 3: p-multigrid143.3127e-09
Euler NACA 0012, p = 3: p-multigrid151.8404e-11

gyre 0.4.0 · 39979fcf90 · · cpu · quick suite · record (JSON)Same code as this build

From a uniform free stream the residual first falls slowly while the pseudo-time step ramps up, then drops by a roughly constant factor per step. The linear systems are solved to a fixed relative tolerance, lin_tol = 10−210^{-2} (inexact Newton with a fixed forcing term), so the terminal convergence is linear at the forcing rate, about two orders per step, not quadratic. The study checks three things per configuration: a residual reduction of ten orders relative to the reference residual, a terminal contraction per step (geometric mean) of at most 5×5 \times lin_tol, and fewer than half the Krylov iterations with p-multigrid. A contraction near the forcing term is the signature of an exact Jacobian; a wrong Jacobian gives a contraction of order one. Both preconditioners solve the same Newton systems — they differ in how much linear work each step costs.

Solver effortNonlinear and linear solver convergence
casepreconditionerlevelsNewton stepsKrylov iterationsmax Krylov / stepconvergedfinal / reference residualterminal contraction (mean)terminal contraction (worst step)wall time [s]
Euler NACA 0012, p = 2block Jacobi2161206182yes2.8557e-120.01092350.013420358.5489
Euler NACA 0012, p = 2p-multigrid2, 1, 01614217yes1.9270e-120.0102030.0154498104.262
Laminar cylinder Re = 40, p = 2block Jacobi2151707320yes1.5243e-120.009804970.0099611148.199
Laminar cylinder Re = 40, p = 2p-multigrid2, 1, 01414319yes8.2062e-110.00763280.0086385373.4588
Euler NACA 0012, p = 3block Jacobi3161360202yes3.3017e-120.01179450.017150274.6616
Euler NACA 0012, p = 3p-multigrid3, 1, 01515423yes5.9042e-110.005823040.0061032479.4693

gyre 0.4.0 · 39979fcf90 · · cpu · quick suite · record (JSON)Same code as this build

Block Jacobi damps element-local error only, so GMRES needs many iterations (and restarts) once the pseudo-time step is large. The p-multigrid V-cycle adds coarse levels down to a finite-volume discretization at p=0p = 0 and reduces the Krylov iteration count; the check requires less than half. That is a ratio of iteration counts, not of work: one multigrid-preconditioned iteration applies the operator several times (smoothing sweeps on every level). The wall times in the table include one-off JIT compilation, which is larger for the multigrid hierarchy. The table also lists the final residual relative to the reference and the terminal contraction, as the mean and for the worst step.

What the iterations cost#

What the Krylov iterations costNonlinear and linear solver convergence
casepreconditionerKrylov iterationsfine-level operator applications per Krylov iterationfine-level operator applications, totalblock solves per iterationwall time, first solve [s]wall time, repeated solve (compiled) [s]repeated solve convergedKrylov iterations, repeated solve
Euler NACA 0012, p = 2block Jacobi120611206158.548981.5204yes1206
Euler NACA 0012, p = 2p-multigrid1427.555561072.8916104.26271.3936yes142
Euler NACA 0012, p = 2p-MG / block Jacobi0.117745—0.889626——0.875776——
Laminar cylinder Re = 40, p = 2block Jacobi170711707148.19964.3311yes1707
Laminar cylinder Re = 40, p = 2p-multigrid1437.555561080.441673.458860.265yes143
Laminar cylinder Re = 40, p = 2p-MG / block Jacobi0.0837727—0.632949——0.936793——
Euler NACA 0012, p = 3block Jacobi136011360174.661671.866yes1360
Euler NACA 0012, p = 3p-multigrid1546.4375991.3751679.469399.4675yes154
Euler NACA 0012, p = 3p-MG / block Jacobi0.113235—0.728952——1.38407——

gyre 0.4.0 · 39979fcf90 · · cpu · quick suite · record (JSON)Same code as this build

Fewer Krylov iterations are not yet less work: one multigrid-preconditioned iteration applies the operator several times. The table therefore gives two measures per solve.

  • Fine-level operator applications. Counted from the definition of the cycle: the Krylov product, the smoothing sweeps and the residual of every level, with coarse levels weighted by their share of the unknowns (Hierarchy.cycle_work). With the default sweeps (2 before, 2 after, 8 on the coarsest level) and the levels 2, 1, 0 in two dimensions, one iteration costs about 7.6 applications where block Jacobi costs one, so the hierarchy has to cut the iteration count by more than that factor before it saves operator work. The weighting overstates the coarse levels and leaves out block solves and transfers: it is a unit for comparing the two preconditioners, not a time.
  • Wall time of the repeated solve. Each solve is run a second time from the free stream when every kernel is compiled, which removes the one-off compilation that dominates the first solve and is larger for the hierarchy.

The rows “p-MG / block Jacobi” give the ratios; a value below one means p-multigrid was cheaper in that measure. The study reports them without a threshold.

Line smoothing on a wall-resolved mesh#

PassedLine-implicit smoothing on a wall-resolved mesh7 of 7 checks passed · 2160 s · quick suite

The turbulent flat plate of the preset flat_plate_sa (Spalart-Allmaras, first cell 5e-5 high) solved with element-block Jacobi and with line Jacobi as the smoother, each on the p-multigrid levels 2-1-0 and on one level, and once more with line Jacobi whose blocks are assembled every second nonlinear step only. Line Jacobi inverts the block-tridiagonal system of every line of strongly coupled elements; the lines follow the thin direction of the cells and are found from the mesh. At the converged flow one forward and one transposed system with the same right-hand side are solved to a relative tolerance of 1e-10. The checks require that the answer does not depend on the preconditioner (drag coefficient, adjoint derivatives), that line Jacobi needs fewer than half the Krylov iterations of block Jacobi on three levels, and that it reaches the tolerance on one level. What block Jacobi does on one level is reported without a threshold, and so are all wall times.

  • flow solves converged with block Jacobi on three levels, with line Jacobi on three levels and on one, and with line Jacobi whose blocks are assembled every second stepyesexpected truepass
  • lines found on the wall-resolved mesh (elements in lines)131expected > 0pass
  • drag coefficient does not depend on the preconditioner (largest relative difference)2.06e-09expected < 1e-6pass
  • forward solve to 1e-10, three levels, both smoothers reaching the tolerance: Krylov iterations, line Jacobi / block Jacobi0.2773expected < 0.5pass
  • transposed (adjoint) solve to 1e-10, three levels, both smoothers reaching the tolerance: Krylov iterations, line Jacobi / block Jacobi0.2834expected < 0.5pass
  • one level, line Jacobi: forward and transposed solves reach the toleranceyesexpected truepass
  • adjoint derivatives of the drag (Mach, Reynolds, free-stream turbulence) from transposed solves that reached 1e-10: largest relative difference between the two smoothers3.16e-09expected < 1e-6pass

Run notes The mesh has 18 lines holding 131 of its 150 elements (the longest 12 elements, 3 colours). On three levels the transposed solve took 70 Krylov iterations with line Jacobi against 247 with block Jacobi (ratio 0.28; wall time of the compiled solve, ratio 0.67); the forward solve 66 against 238. One assembly of the preconditioner on three levels took 4.04 s with line Jacobi against 0.19 s with block Jacobi (the coupling blocks of the line elements are assembled as well, one batch of derivatives per colour). With one level and a tolerance of 1e-10 on the transposed system, block Jacobi reduced the residual by 2.9e-04 in 720 iterations and stopped at the iteration limit; line Jacobi reached the tolerance in 465 iterations. A flow solve assembles the preconditioner at every nonlinear step; the last row of the flow table assembles it every second step only (the repeated solves took 33.1 s (block Jacobi, levels 2-1-0), 233.6 s (line Jacobi, levels 2-1-0), 113.3 s (block Jacobi, one level), 392.5 s (line Jacobi, one level), 147.9 s (line Jacobi, levels 2-1-0, blocks every 2nd step)). Wall times: the column of the linear table is the Krylov solve alone, with a preconditioner that is already assembled; one assembly is timed separately (table 'assembly'). The first flow solve of a configuration includes the one-off compilation. The suite runs its studies in parallel lanes, so other studies were running while these times were taken; they are measurements of small cases on a machine that was not idle, reported without a threshold. Machine load while the study ran: The timings were taken on a busy machine (other processes used 45 % of the capacity of the 8 CPUs while the study ran).

gyre 0.4.0 · 39979fcf90 · · cpu · quick suite · record (JSON)Same code as this build

Linear solves at the converged flow (tolerance 1e-10)Line-implicit smoothing on a wall-resolved mesh
preconditionersystemKrylov iterationsresidual reductiontolerance reachedwall time of the Krylov solve, compiled, preconditioner assembled before [s]
block Jacobi, levels 2-1-0forward2388.9804e-11yes6.18439
block Jacobi, levels 2-1-0transposed (adjoint)2479.4097e-11yes10.9862
line Jacobi, levels 2-1-0forward669.5595e-11yes4.98718
line Jacobi, levels 2-1-0transposed (adjoint)709.4827e-11yes7.34169
block Jacobi, one levelforward7201.1248e-04no4.00668
block Jacobi, one leveltransposed (adjoint)7202.8702e-04no12.8319
line Jacobi, one levelforward4419.9505e-11yes5.62206
line Jacobi, one leveltransposed (adjoint)4658.7870e-11yes5.46822

gyre 0.4.0 · 39979fcf90 · · cpu · quick suite · record (JSON)Same code as this build

The turbulent flat plate of the preset flat_plate_sa: 150 elements of degree 2, the first row 5×10−55\times10^{-5} high, so the cells at the wall are between 400 and about 10 600 times longer than they are thick. The flow is converged with block Jacobi and with line Jacobi as the smoother, each on the p-multigrid levels 2, 1, 0 and on one level, and once more with line Jacobi whose blocks are assembled every second step (solver.precond_lag: 2). At the converged flow one forward and one transposed system with the same right-hand side are then solved to a relative tolerance of 10−1010^{-10}, which is what a direct sensitivity and an adjoint ask for.

The checks require that the answer does not depend on the preconditioner (drag coefficient and three adjoint derivatives agree to 10−610^{-6} relative), that line Jacobi needs fewer than half the Krylov iterations of block Jacobi on three levels, and that it reaches the tolerance on one level. What block Jacobi does on one level is reported as it comes out: it is the stalled solve that made a missed tolerance an error.

Lines of the meshLine-implicit smoothing on a wall-resolved mesh
lineselements in lineselementslongest linecoloursstored line slots
18131150123152

gyre 0.4.0 · 39979fcf90 · · cpu · quick suite · record (JSON)Same code as this build

Flow solve: residual against accumulated Krylov iterationsLine-implicit smoothing on a wall-resolved mesh
Data table
seriesKrylov iterations (accumulated over the Newton steps)|Rw|
block Jacobi, levels 2-1-000.124629
block Jacobi, levels 2-1-020.779736
block Jacobi, levels 2-1-030.425046
block Jacobi, levels 2-1-040.0608727
block Jacobi, levels 2-1-060.0723028
block Jacobi, levels 2-1-080.0489039
block Jacobi, levels 2-1-0100.0260923
block Jacobi, levels 2-1-0130.0125064
block Jacobi, levels 2-1-0180.00622628
block Jacobi, levels 2-1-0240.00437145
block Jacobi, levels 2-1-0290.00336179
block Jacobi, levels 2-1-0360.0029906
block Jacobi, levels 2-1-0440.00305075
block Jacobi, levels 2-1-0530.00231764
block Jacobi, levels 2-1-0650.00194994
block Jacobi, levels 2-1-0820.00157952
block Jacobi, levels 2-1-0990.00119617
block Jacobi, levels 2-1-01160.00110279
block Jacobi, levels 2-1-01360.00103203
block Jacobi, levels 2-1-01560.00116786
block Jacobi, levels 2-1-01760.0013458
block Jacobi, levels 2-1-01960.00156744
block Jacobi, levels 2-1-02150.00225501
block Jacobi, levels 2-1-02340.0019883
block Jacobi, levels 2-1-02530.00208346
block Jacobi, levels 2-1-02710.00171712
block Jacobi, levels 2-1-02900.0019675
block Jacobi, levels 2-1-03080.00130682
block Jacobi, levels 2-1-03276.4408e-04
block Jacobi, levels 2-1-03472.3862e-04
block Jacobi, levels 2-1-03683.7507e-05
block Jacobi, levels 2-1-03961.5333e-06
block Jacobi, levels 2-1-04392.6696e-08
block Jacobi, levels 2-1-04853.2411e-10
block Jacobi, levels 2-1-05402.8753e-12
line Jacobi, levels 2-1-000.124629
line Jacobi, levels 2-1-020.779735
line Jacobi, levels 2-1-030.424873
line Jacobi, levels 2-1-040.0608065
line Jacobi, levels 2-1-060.0722699
line Jacobi, levels 2-1-080.0488743
line Jacobi, levels 2-1-0100.0260662
line Jacobi, levels 2-1-0120.0124835
line Jacobi, levels 2-1-0150.00622118
line Jacobi, levels 2-1-0180.00437353
line Jacobi, levels 2-1-0200.00336289
line Jacobi, levels 2-1-0220.0029882
line Jacobi, levels 2-1-0250.00305218
line Jacobi, levels 2-1-0280.002317
line Jacobi, levels 2-1-0330.00195713
line Jacobi, levels 2-1-0410.00158011
line Jacobi, levels 2-1-0500.00119808
line Jacobi, levels 2-1-0600.00110831
line Jacobi, levels 2-1-0710.00103444
line Jacobi, levels 2-1-0820.00117389
line Jacobi, levels 2-1-0930.0013467
line Jacobi, levels 2-1-01040.00156734
line Jacobi, levels 2-1-01150.00225191
line Jacobi, levels 2-1-01260.0019863
line Jacobi, levels 2-1-01370.00207889
line Jacobi, levels 2-1-01470.00171564
line Jacobi, levels 2-1-01570.00197754
line Jacobi, levels 2-1-01670.00131292
line Jacobi, levels 2-1-01776.4372e-04
line Jacobi, levels 2-1-01882.3866e-04
line Jacobi, levels 2-1-02003.7004e-05
line Jacobi, levels 2-1-02121.4892e-06
line Jacobi, levels 2-1-02272.4781e-08
line Jacobi, levels 2-1-02472.6187e-10
line Jacobi, levels 2-1-02671.9058e-12
block Jacobi, one level00.124629
block Jacobi, one level20.780012
block Jacobi, one level40.424791
block Jacobi, one level70.0607134
block Jacobi, one level160.0724354
block Jacobi, one level260.0488405
block Jacobi, one level410.0261245
block Jacobi, one level640.0125151
block Jacobi, one level990.00623836
block Jacobi, one level1390.0043708
block Jacobi, one level1760.00335876
block Jacobi, one level2200.00298865
block Jacobi, one level2750.00304844
block Jacobi, one level3330.00232512
block Jacobi, one level4020.00221679
block Jacobi, one level4980.0015586
block Jacobi, one level6180.00124381
block Jacobi, one level7380.00110093
block Jacobi, one level8580.00105029
block Jacobi, one level9780.00118134
block Jacobi, one level10980.00134457
block Jacobi, one level12180.00159304
block Jacobi, one level13380.00228624
block Jacobi, one level14580.002025
block Jacobi, one level15780.00214107
block Jacobi, one level16980.00175348
block Jacobi, one level18180.00201657
block Jacobi, one level19380.00133557
block Jacobi, one level20586.4230e-04
block Jacobi, one level21782.2451e-04
block Jacobi, one level22983.0249e-05
block Jacobi, one level24183.1724e-06
block Jacobi, one level25387.8332e-07
block Jacobi, one level26582.8510e-07
block Jacobi, one level27789.9788e-08
block Jacobi, one level28983.3119e-08
block Jacobi, one level30189.3015e-09
block Jacobi, one level31382.4657e-09
block Jacobi, one level32586.7185e-10
block Jacobi, one level33781.7882e-10
block Jacobi, one level34984.2462e-11
line Jacobi, one level00.124629
line Jacobi, one level10.779734
line Jacobi, one level20.423598
line Jacobi, one level30.06012
line Jacobi, one level70.0722277
line Jacobi, one level120.0487814
line Jacobi, one level180.025992
line Jacobi, one level280.0124417
line Jacobi, one level450.00620997
line Jacobi, one level640.0043681
line Jacobi, one level810.00335982
line Jacobi, one level970.00298978
line Jacobi, one level1190.00305384
line Jacobi, one level1430.00230572
line Jacobi, one level1730.00182937
line Jacobi, one level2230.00148708
line Jacobi, one level2810.00144857
line Jacobi, one level3500.0012345
line Jacobi, one level4260.00108126
line Jacobi, one level5050.00113056
line Jacobi, one level5860.00125355
line Jacobi, one level6690.00149991
line Jacobi, one level7510.00220627
line Jacobi, one level8360.00207811
line Jacobi, one level9210.00206307
line Jacobi, one level10040.00166118
line Jacobi, one level10890.00183783
line Jacobi, one level11750.00122033
line Jacobi, one level12656.0889e-04
line Jacobi, one level13612.2528e-04
line Jacobi, one level14603.5600e-05
line Jacobi, one level15631.4746e-06
line Jacobi, one level16812.4610e-08
line Jacobi, one level18016.0836e-10
line Jacobi, one level19145.9566e-12
line Jacobi, levels 2-1-0, blocks every 2nd step00.124629
line Jacobi, levels 2-1-0, blocks every 2nd step20.779735
line Jacobi, levels 2-1-0, blocks every 2nd step40.42343
line Jacobi, levels 2-1-0, blocks every 2nd step50.0604124
line Jacobi, levels 2-1-0, blocks every 2nd step80.0723523
line Jacobi, levels 2-1-0, blocks every 2nd step100.0487729
line Jacobi, levels 2-1-0, blocks every 2nd step120.0259908
line Jacobi, levels 2-1-0, blocks every 2nd step140.0124298
line Jacobi, levels 2-1-0, blocks every 2nd step170.00619779
line Jacobi, levels 2-1-0, blocks every 2nd step200.00436293
line Jacobi, levels 2-1-0, blocks every 2nd step220.00335731
line Jacobi, levels 2-1-0, blocks every 2nd step240.00298214
line Jacobi, levels 2-1-0, blocks every 2nd step270.00305558
line Jacobi, levels 2-1-0, blocks every 2nd step300.00230067
line Jacobi, levels 2-1-0, blocks every 2nd step350.0018162
line Jacobi, levels 2-1-0, blocks every 2nd step430.00148714
line Jacobi, levels 2-1-0, blocks every 2nd step530.00134297
line Jacobi, levels 2-1-0, blocks every 2nd step630.00118022
line Jacobi, levels 2-1-0, blocks every 2nd step740.00105966
line Jacobi, levels 2-1-0, blocks every 2nd step850.00112072
line Jacobi, levels 2-1-0, blocks every 2nd step960.00125874
line Jacobi, levels 2-1-0, blocks every 2nd step1070.00150899
line Jacobi, levels 2-1-0, blocks every 2nd step1180.00220882
line Jacobi, levels 2-1-0, blocks every 2nd step1290.00208793
line Jacobi, levels 2-1-0, blocks every 2nd step1390.00204449
line Jacobi, levels 2-1-0, blocks every 2nd step1490.00164751
line Jacobi, levels 2-1-0, blocks every 2nd step1590.00181937
line Jacobi, levels 2-1-0, blocks every 2nd step1690.00120896
line Jacobi, levels 2-1-0, blocks every 2nd step1796.0570e-04
line Jacobi, levels 2-1-0, blocks every 2nd step1902.2356e-04
line Jacobi, levels 2-1-0, blocks every 2nd step2023.5353e-05
line Jacobi, levels 2-1-0, blocks every 2nd step2141.4646e-06
line Jacobi, levels 2-1-0, blocks every 2nd step2282.5669e-08
line Jacobi, levels 2-1-0, blocks every 2nd step2482.6692e-10
line Jacobi, levels 2-1-0, blocks every 2nd step2682.0002e-12

gyre 0.4.0 · 39979fcf90 · · cpu · quick suite · record (JSON)Same code as this build

One assembly of the preconditioner at the converged flowLine-implicit smoothing on a wall-resolved mesh
preconditionerwall time, compiled, fastest of five [s]
block Jacobi, levels 2-1-00.1873
line Jacobi, levels 2-1-04.04077
block Jacobi, one level3.00486
line Jacobi, one level4.08774

gyre 0.4.0 · 39979fcf90 · · cpu · quick suite · record (JSON)Same code as this build

Flow solve from the free streamLine-implicit smoothing on a wall-resolved mesh
preconditionernonlinear stepsKrylov iterationsconvergeddrag coefficientwall time, first solve [s]wall time, repeated solve (compiled) [s]repeated solve converged
block Jacobi, levels 2-1-034540yes0.0057536586.600633.1422yes
line Jacobi, levels 2-1-034267yes0.00575365159.257233.618yes
block Jacobi, one level403498yes0.00575365136.85113.283yes
line Jacobi, one level341914yes0.00575365245.519392.492yes
line Jacobi, levels 2-1-0, blocks every 2nd step34268yes0.00575365189.424147.94yes

gyre 0.4.0 · 39979fcf90 · · cpu · quick suite · record (JSON)Same code as this build

Adjoint derivatives of the drag coefficient, levels 2-1-0Line-implicit smoothing on a wall-resolved mesh
inputblock Jacobiline Jacobirelative difference
mach-0.0498736-0.04987363.1572e-09
re-3.0651e-10-3.0651e-101.8531e-10
chi_inf3.6050e-063.6050e-061.6364e-09

gyre 0.4.0 · 39979fcf90 · · cpu · quick suite · record (JSON)Same code as this build

In three dimensions#

PassedLine-implicit smoothing in three dimensions6 of 6 checks passed · 620.4 s · quick suite

Steady laminar flow past a sphere (Re = 100, M = 0.2, degree 2) on a cubed-sphere shell whose layers grow by a factor 2.2 from the wall, so that the first layers are several times wider than they are thick. The flow is converged with element-block Jacobi and with line Jacobi as the smoother of the p-multigrid levels 2-1-0; at the converged flow one forward and one transposed system with the same right-hand side are solved to a relative tolerance of 1e-10. The lines run radially through the anisotropic layers of all six blocks of the mesh. The checks require the same drag coefficient with both smoothers, the same adjoint derivatives when the transposed systems are solved to 1e-12 (their difference at 1e-10 is reported), and fewer than half the Krylov iterations with line Jacobi in both linear solves. Wall times are reported without a threshold.

  • flow solves converged with both smoothersyesexpected truepass
  • lines found on the shell with thin wall layers (elements in lines)216expected > 0pass
  • drag coefficient: relative difference between the two smoothers6.58e-11expected < 1e-6pass
  • forward solve to 1e-10, both smoothers reaching the tolerance: Krylov iterations, line Jacobi / block Jacobi0.3286expected < 0.5pass
  • transposed (adjoint) solve to 1e-10, both smoothers reaching the tolerance: Krylov iterations, line Jacobi / block Jacobi0.3464expected < 0.5pass
  • adjoint derivatives of the drag (Mach, Reynolds, angle of attack) from transposed solves that reached 1e-12: largest difference between the two smoothers, relative to the largest derivative8.64e-08expected < 1e-6pass

Run notes The mesh has 54 lines holding 216 of its 432 elements (the longest 4 elements, 3 colours); 58320 unknowns. The transposed solve took 62 Krylov iterations with line Jacobi against 179 with block Jacobi (ratio 0.35; compiled wall time, ratio 0.29), the forward solve 69 against 210. The adjoint derivatives of the two smoothers differ by at most 2.4e-06 of the largest derivative when the transposed systems are solved to 1e-10 (in the derivative with respect to 'mach') and by at most 8.6e-08 at 1e-12. Between the two tolerances the block-Jacobi value of that derivative itself moves by 7.5e-06 of the largest derivative (last column of the table): the difference between the smoothers is the linear tolerance, amplified by the conditioning of that derivative. One assembly of the preconditioner took 1.18 s with line Jacobi against 0.39 s with block Jacobi, and a flow solve assembles it at every nonlinear step: the repeated flow solves took 13.9 s (block Jacobi) and 19.5 s (line Jacobi). The mesh is coarse and its layers are thinner than a laminar boundary layer at this Reynolds number needs: the case measures the solver, not the flow. Wall times: the column of the linear table is the Krylov solve alone, with a preconditioner that is already assembled; one assembly is timed separately (table 'assembly'). The first flow solve of a configuration includes the one-off compilation. The suite runs its studies in parallel lanes, so other studies were running while these times were taken; they are measurements of small cases on a machine that was not idle, reported without a threshold. Machine load while the study ran: The timings were taken on a busy machine (other processes used 74 % of the capacity of the 8 CPUs while the study ran; the GPU was 17 % utilized when the study started).

gyre 0.4.0 · 39979fcf90 · · gpu · NVIDIA GeForce RTX 3060 Ti · quick suite · record (JSON)Same code as this build

Linear solves at the converged flow (tolerance 1e-10)Line-implicit smoothing in three dimensions
preconditionersystemKrylov iterationsresidual reductiontolerance reachedwall time of the Krylov solve, compiled, preconditioner assembled before [s]
block Jacobiforward2109.2582e-11yes7.68143
block Jacobitransposed (adjoint)1799.9328e-11yes5.71461
line Jacobiforward697.6571e-11yes1.86119
line Jacobitransposed (adjoint)627.3926e-11yes1.67616

gyre 0.4.0 · 39979fcf90 · · gpu · NVIDIA GeForce RTX 3060 Ti · quick suite · record (JSON)Same code as this build

Steady laminar flow past a sphere at a Reynolds number of 100, degree 2, on a cubed-sphere shell whose layers grow by a factor 2.2 from the wall: the first layers are several times wider than they are thick, and the lines run radially through them in all six blocks of the mesh, whose local axes are rotated against each other. Elements further out are close to isotropic and stay with block Jacobi. The comparison is narrower than that of the flat plate: the two smoothers on the p-multigrid levels 2, 1, 0 only, with no run on one level and none with lagged blocks, and with blocks of 135×135135\times135 instead of 45×4545\times45.

The checks require the same drag coefficient with both smoothers to 10−610^{-6} relative, fewer than half the Krylov iterations with line Jacobi in the forward and in the transposed solve to 10−1010^{-10}, and adjoint derivatives with respect to Mach number, Reynolds number and angle of attack that agree to 10−610^{-6} of the largest derivative when the transposed systems are solved to 10−1210^{-12}. On the flat plate the derivatives come from solves to 10−1010^{-10} and each must agree to 10−610^{-6} of its own value; why this case is checked differently is explained below.

Lines of the meshLine-implicit smoothing in three dimensions
lineselements in lineselementslongest linecoloursstored line slots
5421643243216

gyre 0.4.0 · 39979fcf90 · · gpu · NVIDIA GeForce RTX 3060 Ti · quick suite · record (JSON)Same code as this build

One assembly of the preconditioner at the converged flowLine-implicit smoothing in three dimensions
preconditionerwall time, compiled, fastest of five [s]
block Jacobi0.391384
line Jacobi1.17592

gyre 0.4.0 · 39979fcf90 · · gpu · NVIDIA GeForce RTX 3060 Ti · quick suite · record (JSON)Same code as this build

Flow solve from the free streamLine-implicit smoothing in three dimensions
preconditionerlevelsnonlinear stepsKrylov iterationsconvergeddrag coefficientwall time, first solve [s]wall time, repeated solve (compiled) [s]repeated solve converged
block Jacobi2, 1, 013193yes1.13546141.4313.8529yes
line Jacobi2, 1, 013123yes1.13546109.47519.5358yes

gyre 0.4.0 · 39979fcf90 · · gpu · NVIDIA GeForce RTX 3060 Ti · quick suite · record (JSON)Same code as this build

Adjoint derivatives of the drag coefficientLine-implicit smoothing in three dimensions
inputblock Jacobi (solve to 1e-12)line Jacobi (solve to 1e-12)difference / largest derivative, solves to 1e-12difference / largest derivative, solves to 1e-10block Jacobi: change from 1e-10 to 1e-12 / largest derivative
mach-0.00635239-0.006352398.6433e-082.4227e-067.4871e-06
re-0.0055249-0.00552496.1495e-112.9472e-102.6955e-10
alpha2.2762e-14-3.4771e-149.0568e-126.3684e-117.2750e-11

gyre 0.4.0 · 39979fcf90 · · gpu · NVIDIA GeForce RTX 3060 Ti · quick suite · record (JSON)Same code as this build

With transposed solves to 10−1010^{-10} the two smoothers give Mach-number derivatives of the drag that differ in the sixth digit, while the Reynolds-number derivatives agree to ten. At a fixed state the drag of this case does not depend on the Mach number (it is the momentum flux through an adiabatic wall), so the derivative is the adjoint-weighted term −ψT ∂Rw/∂M-\boldsymbol\psi^{\mathsf T}\,\partial\mathbf R_w/\partial M alone (adjoint formula). That term is badly conditioned: the Mach number enters the residual through the far-field state, whose pressure 1/(γM2)1/(\gamma M^2) changes by −2/(γM3)-2/(\gamma M^3) per unit Mach number, and at the low Mach number of this case the flow answers mostly with a shift of its pressure level, which exerts no net force on a closed body. The derivative of the drag is the small remainder of that large response. An error of 10−1010^{-10} in the adjoint is amplified accordingly, with either smoother: the last column shows that the block-Jacobi value itself moves by as much when the tolerance is tightened. At 10−1210^{-12} the two agree, which is what the check requires. For a design loop at low Mach number the consequence is the same as everywhere else in Gyre: the tolerance of totals is the accuracy of the derivative, scaled by its conditioning.

Single-precision blocks#

PassedSingle-precision blocks in the preconditioner6 of 6 checks passed · 2649 s · quick suite

Each case is solved twice from the free stream, with the blocks of the smoother in double and in single precision (solver.precond_precision), and at the converged flow one forward and one transposed system with the same right-hand side are solved to a relative tolerance of 1e-10. A rounded preconditioner must change neither the answers nor, to speak of, the work: the checks require the same drag coefficient, the same solutions of the linear systems, at most a tenth more Krylov iterations, and half the memory for the stored blocks. The cases cover turbulent flow, whose blocks are the worst conditioned, with both smoothers, and laminar flow in three dimensions.

  • every flow solve and every linear solve converged, in both precisionsyesexpected truepass
  • drag coefficient: largest relative difference between the precisions2.25e-14expected < 1e-08pass
  • solutions of the linear systems (tolerance 1e-10): largest relative difference between the precisions4.47e-11expected < 1e-07pass
  • Krylov iterations of the flow solve, single / double: largest ratio1expected < 1.1pass
  • Krylov iterations of the linear solves, single / double: largest ratio1expected < 1.1pass
  • memory of the stored blocks, single / double: largest ratio0.5expected < 0.51pass

Run notes With solver.precond_precision 'single' the columns of the element blocks (and of the coupling blocks of the line smoother) are cast to float32 as they are computed, the inverses and the factorisation of the line systems are computed in float64 from them and rounded, and the blocks are applied in float32. The residual, its Jacobian products and the Krylov vectors are float64 throughout, so the converged flow and the solutions of the linear systems are those of the same equations: the differences in the table are the tolerances of the solves. The cases are small; what the option is for, a problem that does not fit the device otherwise, is in the study scale_3d.

gyre 0.4.0 · 39979fcf90 · · cpu · quick suite · record (JSON)Same code as this build

Double against single precision of the smoother's blocksSingle-precision blocks in the preconditioner
caseblocksunknownsnonlinear stepsKrylov iterations, flow solvedrag coefficientKrylov iterations, forward solveKrylov iterations, transposed solvelinear solutions: relative difference to doublestored blocks [MB]
turbulent flat plate (SA)double6750355400.00575365238249—2.94
turbulent flat plate (SA)single6750355400.005753652382491.7075e-111.47
turbulent flat plate (SA), line smootherdouble6750352670.005753656663—8.8984
turbulent flat plate (SA), line smoothersingle6750352670.0057536566634.4650e-114.4492
laminar sphere (3-D), Re = 100, degree 1double384014792.134864643—1.248
laminar sphere (3-D), Re = 100, degree 1single384014792.1348646434.1034e-140.624

gyre 0.4.0 · 39979fcf90 · · cpu · quick suite · record (JSON)Same code as this build

solver.precond_precision: single keeps the blocks of the smoother (the element-block inverses and the factors of the line systems) in single precision (how). The residual, its Jacobian products and the Krylov vectors stay in double precision, so the equations that are solved are the same, and a preconditioner that is rounded in the eighth digit should cost no iterations either. Each case is solved twice from the free stream and then one forward and one transposed system are solved to 10−1010^{-10} at the converged flow. The checks require the same drag coefficient, the same solutions of the linear systems, at most a tenth more Krylov iterations and half the memory of the stored blocks. The turbulent flat plate is the hard case for a rounded block (its turbulence rows are scaled by six orders against the mean-flow rows) and is run with both smoothers; a small laminar sphere adds three dimensions.

The cases are small. What the option is for, a problem that does not fit the device with double-precision blocks, is recorded with the largest 3-D solve.

Inside a session each solve restarts from the previous converged state in the Newton regime. The service test-suite asserts that a re-solve after a small change of angle of attack needs fewer nonlinear iterations than the cold start. A solve that does not converge does not replace that state: the next solve warm-starts from the last converged one (tests/test_discipline.py::test_failed_solve_keeps_last_converged_state).

    ↑ ↓ moveEnter open[ ] change depthResults show section and depth level