Skip to main content

Learning with Gauss–Newton curvature

Curvature methods have different mathematical objects and calling contracts. Gauss–Newton uses a residual Jacobian and approximates least-squares curvature. It is not the full Hessian of an arbitrary loss.

A functional optimizer​

omnibias.torch.optim.GaussNewton takes a residual function and a flat parameter vector. Its step returns a new vector and diagnostic information. It is not interchangeable with a torch.optim.Optimizer closure.

curvature.py
"""GaussNewton is functional: step returns a new flat parameter vector."""
import torch
from omnibias.torch.optim import GaussNewton

x = torch.tensor([-1.0, 0.0, 1.0], dtype=torch.float64)
y = 2 * x + 1

def residual(parameters):
    return parameters[0] * x + parameters[1] - y

parameters = torch.zeros(2, dtype=torch.float64)
optimizer = GaussNewton(solver="qr", damping=1e-3)
before = residual(parameters).square().mean()
for _ in range(4):
    parameters, info = optimizer.step(residual, parameters)
after = residual(parameters).square().mean()
assert after < before
assert torch.isfinite(parameters).all()
torch.testing.assert_close(parameters, torch.tensor([2.0, 1.0], dtype=torch.float64),
                           rtol=1e-6, atol=1e-6)
print("Recovered slope and intercept:", parameters.detach().tolist())

Download the complete script.

This two-parameter least-squares problem recovers a slope and intercept. The assertions check that the residual decreases and the known solution is recovered. They do not establish an optimizer ranking on PDEs or large networks.

The QR solve works on the least-squares system; normal-equation solves square its condition number. Matrix-free alternatives have separate tolerances and iteration limits. Consult Gauss–Newton and other optimizer contracts.

For closed-form parameter curvature on supported one-layer fields, see omnibias-curvature, an AGPL-or-commercial package. For full-model curvature, distinguish exact, structured, and stochastic paths.

Solve a complete problem​

Continue with the executed notebook, including full initialization, training or optimization, numerical assertions, plots, and recorded results.