Activation dictionary

Each backend activation carries metadata: forward(z), derivative(z), fastpath(z, n), an optional riccati_polynomial, a noise_model (the GLM family for which σ is the log-partition), and an operator_role (what the K=2 collapse computes).

from omnibias.torch import list_activations, get_activation
print(list_activations())
spec = get_activation("tanh")

Smooth (Riccati) family

A closed-form derivative tower is available at every order. This is the default choice for PINN-style architectures and anything needing higher derivatives.

name	σ(z)	σ'(z)	Fast path
sigmoid	1/(1 + e^-z)	s(1 - s)	Eulerian
tanh	tanh(z)	1 - t²	Legendre-style
softplus	log(1 + e^z)	sigmoid(z)	reuses Eulerian
gaussian	exp(-z²/2)	-z·exp(-z²/2)	Hermite

Proximal family

Designed so that the K=2 bias-collapse output is a classical proximal operator. Fast paths defined for n ∈ {0, 1} (the orders these use cases call for); higher n raises NotImplementedError.

name	K=2 collapse output	Operator role
huber	clip(z, -τ, τ)	proximal of the L1 norm (ISTA soft-shrink)
arctan	1/(1 + z²)	Cauchy IRLS weight
log1pu2	2z/(1 + z²)	redescending M-estimator (Black–Anandan)

Build a Huber spec with a custom threshold via omnibias.torch.activations.proximal.make_huber_spec(tau=...).

Classical family

Drop-in compatibility with existing pretrained backbones. Fast paths for n ∈ {0, 1} (and all n for exp).

name	σ(z)	σ'(z)
exp	exp(z)	exp(z) (eigenfunction of d/dz)
relu	max(z, 0)	Heaviside step (PyTorch convention H(0)=0)
silu	z·σ(z)	σ + z·σ·(1 - σ)
gelu	z·Φ(z) (exact)	Φ(z) + z·φ(z)

For relu, silu, and gelu, only op="identity" and op="grad" are valid (no higher-order fast path).

Choosing a base activation

Ask, in order:

1. What operator role do I need? Match the K=2 collapse output.
2. What derivative orders do I need? Smooth family for unrestricted orders; classical / proximal for n ≤ 1.
3. What noise model does the upstream loss assume? Match σ to the GLM family whose log-partition it is.
4. What inductive bias at init? Lemma-1 init makes the layer behave as the base σ at step zero.

See Choosing an activation for a decision guide with worked examples.

Adding a custom activation

import torch
from omnibias.core import ActivationSpec
from omnibias.torch.activations import register_activation

spec = ActivationSpec(
    name="my_swish",
    forward=lambda z: z * torch.sigmoid(z * 1.7),
    derivative=None,         # mark unavailable
    fastpath=None,
    riccati_polynomial=None,
    noise_model="none",
    operator_role="custom; not a known proximal or GLM family",
)
register_activation(spec)

After registration, get_activation("my_swish") and OMBU(..., base="my_swish") work as for any built-in. Operator paths that require a fast path will reject specs without one with a clear error.

Per-activation order limits

The exact maximum supported order per activation is recorded in the Stability matrix. Do not assume an arbitrary order is implemented for non-Riccati activations.