API REFERENCE/omnibias-torch

omnibias.torch.activations.classical

0.5.0rc1Release candidatePython >=3.10Apache-2.0

PyTorch activation derivatives, trainable operator layers, network jets, and direct differential-operator contractions.

import omnibias.torch.activations.classical

Module contract & mathematical conventions

Classical activations: relu, gelu, silu, exp.

All four now carry closed-form all-orders fast paths:

• exp -- every derivative equals exp(z).
• silu = z * sigmoid(z) and gelu = z * Phi(z) -- **exact** all orders via Leibniz on the analytic z * f(z) product, reusing the sigmoid / Gaussian-Hermite towers.
• relu -- closed-form on the **almost-everywhere / regular-part** convention: n = 0 value, n = 1 Heaviside, n >= 2 zero away from the kink (the singular delta at z = 0 is dropped). Its smooth beta-tempered twin soft_relu lives in omnibias.torch.activations.tempered.

Shared formulas do not imply universal floating-point identity. Verify dtype, shape, derivative-order support, and the stated numerical assumptions for the path you use.

Module source ↗Source revision 9a2d73df0573Running reference examples →

CONSTANT

EXP

omnibias.torch.activations.classical.EXPSource ↗

EXP = register_activation(ActivationSpec(name='exp', forward=_exp_forward, derivative=_exp_derivative, fastpath=_exp_fastpath, integral=_exp_forward, riccati_polynomial=(0.0, 1.0), noise_model='poisson', operator_role='K=2 collapse -> exp(z) (eigenfunction of d/dz); log-link Newton step for Poisson regression.', limit_neg_inf=0.0))

This exported constant is defined by the expression above. Its exact value and annotation come from the pinned source; use it with the module contract.

Read the implementation

EXP = register_activation(
    ActivationSpec(
        name="exp",
        forward=_exp_forward,
        derivative=_exp_derivative,
        fastpath=_exp_fastpath,
        integral=_exp_forward,
        riccati_polynomial=(0.0, 1.0),  # dE/dz = E, so P(E) = E
        noise_model="poisson",
        operator_role=(
            "K=2 collapse -> exp(z) (eigenfunction of d/dz); "
            "log-link Newton step for Poisson regression."
        ),
        limit_neg_inf=0.0,  # exp -> 0 as z -> -inf; diverges as z -> +inf
    )
)

Related APIsomnibias.core.ActivationSpec↗omnibias.torch.register_activation↗

CONSTANT

GELU

omnibias.torch.activations.classical.GELUSource ↗

GELU = register_activation(ActivationSpec(name='gelu', forward=_gelu_forward, derivative=_gelu_derivative, fastpath=_gelu_fastpath, integral=_gelu_integral, riccati_polynomial=None, noise_model='none', operator_role='K=2 collapse -> Phi(z) + z * phi(z); smoothed half-space gate, transformer FFN compatibility.'))

This exported constant is defined by the expression above. Its exact value and annotation come from the pinned source; use it with the module contract.

Read the implementation

GELU = register_activation(
    ActivationSpec(
        name="gelu",
        forward=_gelu_forward,
        derivative=_gelu_derivative,
        fastpath=_gelu_fastpath,
        integral=_gelu_integral,
        riccati_polynomial=None,
        noise_model="none",
        operator_role=(
            "K=2 collapse -> Phi(z) + z * phi(z); "
            "smoothed half-space gate, transformer FFN compatibility."
        ),
    )
)

Related APIsomnibias.core.ActivationSpec↗omnibias.torch.register_activation↗

CONSTANT

RELU

omnibias.torch.activations.classical.RELUSource ↗

RELU = register_activation(ActivationSpec(name='relu', forward=_relu_forward, derivative=_relu_derivative, fastpath=_relu_fastpath, integral=_relu_integral, riccati_polynomial=None, noise_model='none', operator_role='K=2 collapse -> Heaviside step; equality-constraint indicator, Moore-Penrose pseudoinverse limit.'))

This exported constant is defined by the expression above. Its exact value and annotation come from the pinned source; use it with the module contract.

Read the implementation

RELU = register_activation(
    ActivationSpec(
        name="relu",
        forward=_relu_forward,
        derivative=_relu_derivative,
        fastpath=_relu_fastpath,
        integral=_relu_integral,
        riccati_polynomial=None,
        noise_model="none",
        operator_role=(
            "K=2 collapse -> Heaviside step; equality-constraint indicator, "
            "Moore-Penrose pseudoinverse limit."
        ),
    )
)

Related APIsomnibias.core.ActivationSpec↗omnibias.torch.register_activation↗

CONSTANT

SILU

omnibias.torch.activations.classical.SILUSource ↗

SILU = register_activation(ActivationSpec(name='silu', forward=_silu_forward, derivative=_silu_derivative, fastpath=_silu_fastpath, riccati_polynomial=None, noise_model='none', operator_role='K=2 collapse -> sigmoid + z * sigmoid * (1 - sigmoid); smoothed gate for transformer FFN compatibility.', aliases=('swish',)))

This exported constant is defined by the expression above. Its exact value and annotation come from the pinned source; use it with the module contract.

Read the implementation

SILU = register_activation(
    ActivationSpec(
        name="silu",
        forward=_silu_forward,
        derivative=_silu_derivative,
        fastpath=_silu_fastpath,
        riccati_polynomial=None,
        noise_model="none",
        operator_role=(
            "K=2 collapse -> sigmoid + z * sigmoid * (1 - sigmoid); "
            "smoothed gate for transformer FFN compatibility."
        ),
        aliases=("swish",),
    )
)

Related APIsomnibias.core.ActivationSpec↗omnibias.torch.register_activation↗

← Package overviewRunnable learning examples →