Benchmark methodology
A benchmark number is only meaningful with its method attached. This page is the method; results are the numbers.
Tiers
- Smoke tier runs on a laptop CPU in seconds. It verifies correctness and
that the scaling shape is right (e.g. Laplacian flat in
D). Use it to reproduce the trend. - Full tier runs on an accelerator and produces the published wall-clock and memory numbers.
Baselines
omnibias is compared against the honest autodiff alternatives:
- Dense Hessian trace —
jax.hessianthen trace, for the Laplacian. - Forward-Laplacian (folx-style) — the structured forward pass.
- Nested autodiff — for iterated Laplacians
Δᵏ.
The baseline is always the method a careful practitioner would actually use, not a strawman.
Rules that make numbers honest
Benchmark honesty rules
- Answers must match. Every comparison reports the max abs difference; it
must be
≤ 1e-15in float64. A speedup against a wrong answer is not reported. - Report precision. float64 for correctness; float32 only when the application tolerates it, and labelled.
- Report the hardware tier (memory class), never a vendor or hostname.
- Warm up and block. Exclude compile time; block on results so dispatch isn't mistaken for compute.
- Repeat and report spread, not a single lucky run.
Running the smoke tier
JAX_PLATFORMS=cpu python -m bench.laplacian_scaling.dimension_sweep \
--dims 3 12 30 --hidden 32 --batch 64 --repeats 3 --seed 0
What we measure
| Quantity | Why |
|---|---|
Per-sample Laplacian time vs. D | shows O(1)-in-D behavior |
Peak memory vs. D | shows no quadratic Hessian materialization |
Δᵏ time vs. k | shows flat-in-order cost |
| max abs error vs. baseline | shows the answer is identical |