Complexity
The benchmark numbers are not luck; they fall out of the complexity of the closed-form path. This page is the "why."
Setup
Consider a one-layer scalar field over x ∈ ℝᴰ with hidden width H:
f(x) = Σ_j c_j · σ(w_jᵀx + β_j) + b
The Laplacian
The Laplacian is:
Δf(x) = Σ_j c_j · σ''(w_jᵀx + β_j) · ‖w_j‖²
Two facts drive the win:
σ''is one closed-form tower evaluation — not a second autodiff pass.‖w_j‖²does not depend onx— it is computed once per neuron and reused for every input in the batch.
So once the pre-activations are formed (the cost you already pay for the forward
pass), the Laplacian adds an O(H) contraction that is independent of D.
Versus dense-Hessian autodiff
A dense Hessian builds a D × D object and differentiates the graph to form it:
O(D²) memory and a build cost that grows with D. Tracing it for the
Laplacian does not avoid materializing it. That is the quadratic curve in the
results.
Iterated Laplacians Δᵏ
Each additional Laplacian order needs higher even derivatives σ^(2k), which
the tower produces at the same cost per order — the recurrence is flat in
n. Nested autodiff instead stacks another differentiation pass each time, so
its graph (and memory) grows until it OOMs. Hence:
omnibias Δᵏ: O(k) towers, each O(H) → ~flat wall-clock
nested autodiff Δᵏ: graph grows with k → ~480× at k=3, OOM at k=4
The headline, summarized
| Quantity | omnibias | dense-Hessian autodiff |
|---|---|---|
| Laplacian, per sample | O(H), flat in D | grows with D, O(D²) memory |
Δᵏ | O(k) towers | graph grows with k |
| Answer | identical | identical |
The speedup is not a tuned kernel trick — it is the consequence of replacing "differentiate a graph again" with "evaluate one more closed-form coefficient." That is why it is robust across hardware.