
Mosaic benchmarks differentiable physics solvers on accuracy, gradient quality and cost, in compute and in development effort.
What Mosaic measures
If you train a model or run an optimizer through a simulation, the solver has to return a correct prediction (the forward pass) and a usable gradient of that prediction with respect to your inputs (the vector–Jacobian product, VJP). A solver that predicts accurately is of little use for gradient-based work if its VJP is wrong, noisy or ill-conditioned, so Mosaic checks both. The results answer two questions for every solver on every setup:
Is the solver accurate, and are its gradients usable?
Forward accuracy against a reference, gradients against finite differences, and whether an optimizer makes progress through the solver.
At what compute and development cost?
Wall-clock time and peak memory of the forward and VJP calls, and the effort it took to make the solver run the setup.
Comparisons of solvers rarely report what it took to make each one run a problem, from setting a documented option to patching the library. Mosaic records every such intervention, with its reason and evidence, and rates each solver’s effort per setup from L0 (the public API with documented options) to L3 (changes to the solver itself). The solver reference defines the levels and lists every intervention.
Setups
Results are reported per setup, meaning one problem statement with its own experiments and checks. A solver that handles one setup need not handle another, so Mosaic never averages across setups. Each card links to the setup’s results page, and its chips link to the solvers that run it. The badge names the physics, as in the paper: heat transfer (H), structural mechanics (S), and incompressible flow in 2D (F2) and 3D (F3).
Recover the initial velocity field of a 3D flow from a later snapshot, with one control per velocity component and cell of a \(16^3\) grid (12,288 values). Also measures how far each solver’s gradient stays usable as the rollout lengthens.
Terminology
The results and the rest of these docs use five terms with a fixed meaning.
- Domain
-
A family of solvers that share one canonical schema and one directory under
mosaic/tesseracts/. There are three:navier-stokes-grid,structural-meshandthermal-mesh. - Problem
-
The benchmark package selected with
mosaic run -p, which registers experiments and their settings. There are four:ns-grid,ns-3d-grid,structural-meshandthermal-mesh. Both Navier–Stokes problems use thenavier-stokes-griddomain. You only need this term to run the benchmark or contribute to it. - Setup
- A problem statement that results are reported for, such as channel flow past a cylinder. There are six, shown above. A problem can have several setups, and every experiment belongs to exactly one. Checks, effort and deviations are reported per setup.
- Solver
-
One Tesseract under a domain directory, wrapping one library. A solver can serve several problems and setups. The
navier-stokes-gridsolvers serve both Navier–Stokes problems. - Backend
-
The runtime or AD framework a solver is built on, such as JAX or PyTorch, as recorded in its
metadata.mosaic.backend.
Solvers as Tesseracts
Each solver is wrapped in a Docker container by tesseract-core, which exposes a uniform apply / vjp interface. The harness calls those containers as JAX functions through tesseract-jax and never inspects how a solver computes its gradient, so it can compare solvers built on JAX, PyTorch, Julia or hand-written C++ adjoints without a shared dependency. The same containers work outside the benchmark: Use a solver in your code shows how to call one from JAX or PyTorch.
Benchmark suites
Experiments are grouped into suites, which you select with mosaic run --suites. The settings of every experiment (grid sizes, sweep ranges, finite-difference steps, iteration caps and pass criteria) live next to it in the problem configurations under mosaic/benchmarks/problems/, and each results page describes the experiments of its setup.
Forward accuracy (forward). Resolution sweeps against a reference: an analytic solution where one exists (the 2D Taylor–Green vortex, the laminar Kolmogorov shear flow), the Schäfer–Turek literature value for the cylinder, converged high-resolution spectral runs for the 3D Taylor–Green vortex and the turbulent Kolmogorov flow, and for the mesh setups the trimmed mean of all solvers, stored as a precomputed reference. Where it applies, the suite also checks physical laws such as a divergence-free velocity field.
Gradient accuracy (gradient). Central finite differences through the Tesseract interface are the solver-agnostic ground truth. Along random directions \(v_k\), the harness compares the VJP’s directional derivatives \(d^\text{ad}_k = \langle g, v_k \rangle\) of a scalar loss with finite-difference estimates \(d^\text{fd}_k\), at the step size \(\varepsilon\) that fits best, and reports:
- direction agreement, the cosine similarity of \(d^\text{ad}\) and \(d^\text{fd}\) (\(1\) is identical);
- magnitude agreement, the median of \(|d^\text{ad}_k - d^\text{fd}_k| / \max(|d^\text{ad}_k|, |d^\text{fd}_k|)\) (\(0\) is exact).
Depending on the setup, the suite also compares against analytic gradients, follows the gradient error as the rollout lengthens, differentiates through a chain of solver calls, and inspects the singular values of the Jacobian, whose spread shows how ill-conditioned the gradient is.
Optimization (optimization). A gradient-based optimizer (L-BFGS or Adam, depending on the setup) runs through the solver up to an iteration cap. The check passes when the objective falls below a set fraction of its initial value, and drag minimization also requires the final drag to land close to the other solvers’ median. This catches solvers whose gradients pass the finite-difference check but still fail to drive a full optimization.
Cost (cost). Forward and VJP wall-clock time and peak memory, swept over problem size and rollout length on each solver’s hardware target. Each sweep point starts with an untimed warm-up call, which covers compilation, followed by repeated timed trials.
Initial conditions (ics). Generates and plots the initial conditions, without running a solver.
A run writes metrics as JSON, fields as NPZ, and plots as PNG with a Vega-Lite .vl.json spec next to each, under mosaic-results/<problem>/<suite>/<experiment>/. Reading the output explains the layout.