Evaluation
omnibias is evaluated along four independent axes. A release is only as good as the weakest one, so each has its own gate.
The four axes
1. Correctness
Closed-form derivatives are checked against an independent ground truth —
typically autodiff of the same function — and must agree to float64
round-off (≤ 1e-15). This is a correctness check: the closed form and
autodiff compute the same mathematical object two different ways.
2. Parity
The same (activation, order) pair must be bit-identical across PyTorch,
JAX, and Keras 3. See cross-backend parity.
3. Soundness
Every rigorous enclosure must contain a dense deterministic grid and a random sample of true values. An enclosure that excludes a true value is unsound and blocks release. See the certified register.
4. Performance
Speed and memory versus autodiff baselines, always reported with the precision and hardware tier, and always with matching answers. See benchmarking.
The metrics that matter
| Axis | Metric | Target |
|---|---|---|
| Correctness | max abs error vs. ground truth | ≤ 1e-15 (float64) |
| Parity | max ULP difference across backends | 0 (float64) |
| Soundness | fraction of true samples enclosed | 1.0 |
| Performance | speedup + memory ratio vs. autodiff | reported, not cherry-picked |
A speedup only counts if the answer matches. Always pair a performance number with the correctness check that shows the result is identical to the trusted baseline.
See also
- Evaluation gates — these axes, automated.
- Benchmark methodology.