Run the benchmark

This page covers installing Mosaic, running the benchmark and reading its output. Running it locally reproduces the results on your own hardware, which gives numbers that reflect your setup. To contribute a solver or a change, start with the contributing guide once you have a first run working.

Requirements

Mosaic runs every solver inside a Docker container. To run all of them, you need Linux (x86-64) with Docker Engine, an NVIDIA GPU and the NVIDIA Container Toolkit, with Docker configured to use the nvidia runtime. Without a GPU, --gpus none restricts a run to the CPU-only solvers, and mosaic run skips the GPU solvers automatically when it detects no GPU.

macOS (including Apple Silicon) and Windows are not supported, because Docker Desktop can’t pass an NVIDIA GPU through to containers, and not every solver image is known to build on ARM hosts. mosaic run prints a warning when it detects an unsupported platform. On Windows, the closest setup to a Linux host is Docker Engine and the NVIDIA Container Toolkit inside a WSL 2 distribution, though our CI does not cover it.

You also need:

Python ≥ 3.11.
Disk space for the solver images, which range from about 1 GB to more than 10 GB each and take dozens of GB together.
Memory that grows with the problem size. The largest 3D cost sweeps deliberately go past the memory of a 16 GB GPU, so out-of-memory errors at the top of those sweeps are expected and show up as resource limits in the results.

Install

git clone https://github.com/pasteurlabs/mosaic
cd mosaic

# with uv
uv sync

# or with pip
pip install -e .

With uv, either activate the environment (source .venv/bin/activate) or prefix the commands on this page with uv run. For a development setup with tests and linting, see Setup in the contributing guide.

Check the installation:

mosaic --help
mosaic status        # lists every problem and solver, without Docker

First run

This command builds the CPU-only heat-transfer solvers and runs the forward suite at reduced size, which works without a GPU:

mosaic run -p thermal-mesh --suites forward --gpus none --debug

--gpus none keeps only the solvers that don’t use a GPU (deal.II, FEniCS and Firedrake for this problem), and --debug shrinks grid sizes, rollout lengths and sweeps for fast iteration. At the end, mosaic run prints a summary table with one row per problem and one column per suite. Debug results go to directories with a _debug suffix, which mosaic status ignores, so they never mix with full-size results.

Full run

mosaic run

Without options, mosaic run builds every solver image and runs every suite of every problem.

Warning

A full run can take many hours on a machine with a modern GPU, and much longer on CPU only. Consider starting with a single problem (mosaic run -p thermal-mesh) to validate your setup before committing the time and disk space.

If you have several GPUs, --gpus 0,1 runs one solver on each listed GPU in parallel. Without --gpus, solvers run one at a time, and each container can see every GPU.

Choosing what runs

Four options narrow a run, and they combine freely:

Option	Selects
`-p`, `--problems`	Comma-separated problems: `ns-grid`, `ns-3d-grid`, `structural-mesh`, `thermal-mesh`, or `all` (default)
`--suites`	Comma-separated suites: `forward`, `cost`, `gradient`, `optimization`, `ics`, or `all` (default)
`-e`, `--experiments`	One suite (`gradient`), one experiment (`gradient/fd_check`), or one initial condition of an experiment (`forward/agreement/tgv`)
`-s`, `--solvers`	Solvers by display name, as a flat list or a per-problem map (see below)

For example:

mosaic run -p ns-grid --suites forward               # forward suite of ns-grid
mosaic run -p ns-grid -e gradient/fd_check --debug   # one experiment, reduced size
mosaic run -p structural-mesh,thermal-mesh --suites cost

-s accepts two forms:

# Flat list: each problem keeps the listed solvers it has, and problems
# with none of them are skipped.
mosaic run -s OpenFOAM,XLB,deal.II,JAX-FEM

# Per-problem map: problems not in the map keep all their solvers.
mosaic run -s "ns-grid=XLB,jax-cfd;structural-mesh=Firedrake,JAX-FEM"

Names are the display names from the solver reference, matched case-insensitively, so xlb and XLB both work. If a name in a flat list matches no solver in any problem, the run aborts with a “Did you mean…?” hint before any image is built. In a per-problem map, unknown names are skipped with a warning.

A few more options change how a run behaves:

Option	Effect
`--debug`	Smaller grids, shorter rollouts and fewer sweep points
`--gpus <ids>`	Run one solver per listed GPU in parallel, or `none` for CPU-only solvers
`--no-build`	Use the existing images instead of building them
`--no-plots`	Skip plot generation
`--plots-only`	Regenerate plots from existing results without running any solver
`-o`, `--output-dir`	Write results somewhere other than `./mosaic-results` (also settable with `MOSAIC_RESULTS_DIR`)
`-j`, `--jobs`	Number of images built concurrently

mosaic run --help lists every option.

Re-running and resuming

After a run, --only re-runs just the cells (one experiment for one solver) in the given states and leaves the others untouched:

mosaic run --only failed                  # re-run failed cells
mosaic run --only failed,stale            # plus cells whose code changed since they ran
mosaic run --only missing                 # cells that have never run
mosaic run -s PhiFlow --only failed       # one solver's failed cells

The states are failed, anom (flagged by a check), missing, stale and excluded. Combine several with commas, and narrow further with -p, --suites, -e and -s.

If a long run is killed partway through, for example by an out-of-memory error or a reboot, --continue resumes it by skipping experiments whose result.json already exists. Within an experiment that was interrupted, it also skips the solvers already recorded in its result_partial.json, which the harness writes after each solver finishes.

mosaic run --continue                     # resume everything
mosaic run -p ns-grid --continue          # resume one problem
mosaic run --only failed --continue       # retry failed cells, keep the rest

Reading the output

Results are organized by problem, suite and experiment:

mosaic-results/
  ns-grid/
    forward/
      baseline/
        result.json        # metrics per solver and sweep point
        params.json        # settings the experiment ran with
        fields.npz         # output fields
        *.png, *.vl.json   # plots, each with its interactive Vega-Lite spec
      agreement/tgv/       # one directory per initial condition, where there are several
      cylinder/
    gradient/
      fd_check/
    cost/
      spatial_cost/
      vjp_cost/by_N/
    optimization/
      drag_opt/
    ics/
      tgv/                 # the initial condition and its plot
  ns-3d-grid/
  structural-mesh/
  thermal-mesh/

mosaic status summarizes what has run and how each cell did:

mosaic status                             # one table per problem
mosaic status -p ns-grid -f               # one problem, with the reason for every failure
mosaic status --format md > report.md     # Markdown report
mosaic status --format json > snap.json   # machine-readable snapshot

The results overview explains what the checks and states mean.

Troubleshooting

Container start failures

If a solver’s container fails to start, for example because of a broken image, a missing CUDA runtime or an import error in its API module, the solver shows up as a failed cell, and mosaic status -f gives the underlying error. Once the container is fixed, retry with mosaic run --only failed.

When a container dies during a run, the failure reason in mosaic status -f includes how the container exited and the last lines of its log, which are the first place to look for the cause.

“No known GPU vendor” errors

If GPU solvers fail with failed to discover GPU vendor from CDI: no known GPU vendor found (or could not select device driver "" with capabilities: [[gpu]] on older Docker versions), Docker has no NVIDIA runtime that can give the container a GPU. Either install the NVIDIA Container Toolkit and configure Docker to use it, or run only the CPU-only solvers with mosaic run --gpus none.

Container build failures

If tesseract build fails:

Check that Docker is running: docker info
Check disk space: docker system df
For GPU solvers, check the NVIDIA runtime: docker run --rm --gpus all nvidia/cuda:12.2.0-base-ubuntu22.04 nvidia-smi
Build the solver on its own with more output: tesseract --loglevel debug build <solver directory>
Look through the build log for missing system dependencies, which are most common with FEniCS and Firedrake

Solver timeouts

A solver call that runs longer than the HTTP timeout (1200 s by default, set with MOSAIC_TESSERACT_TIMEOUT) is stopped. This happens most often with CPU-only solvers at large grid sizes and with long rollouts on slow solvers. Use --debug for smaller problem sizes, or -s <solver> to isolate one solver.

Memory issues

3D fluid problems at high resolution can exceed the available RAM or VRAM. Out-of-memory kills show up in dmesg, in docker logs and in the failure reason in mosaic status -f.

With --gpus 0,1, solvers run in parallel and their containers share the host’s RAM. To reduce memory pressure, list fewer GPUs or omit the flag, and use --debug for smaller problem sizes. Each container’s memory is capped at a fraction of the host’s RAM (90% by default), which you can change with MOSAIC_CONTAINER_MEM_FRACTION, or remove by setting it to 0.

Getting help

If you run into problems not covered here, visit the Tesseract Forum or open an issue on GitHub.