omnibias.torch.activations.tempered
PyTorch activation derivatives, trainable operator layers, network jets, and direct differential-operator contractions.
import omnibias.torch.activations.temperedModule contract & mathematical conventions
Beta-tempered smooth surrogates of the hard non-smooth activations.
Where omnibias.torch.activations.piecewise gives the *hard* activations
their almost-everywhere tower (dropping the singular part), this module gives
the complementary *smooth* family: differentiable surrogates with a temperature
beta that sharpen to the hard activation as beta -> inf, and whose
higher-order bumps *become* the dropped Dirac deltas in that limit.
Each surrogate reuses an existing closed-form tower through the backend-neutral
omnibias.core.spec.tempered combinator (or a small hand-written Leibniz
product), so every order is exact and algebraically corresponding across backends:
============ ================================== =========================
Surrogate definition hard limit (beta -> inf)
============ ================================== =========================
soft_relu softplus(beta z) / beta relu
soft_step sigmoid(beta z) Heaviside
soft_sign tanh(beta z) (= smooth_sign) sign
soft_abs sqrt(z^2 + eps^2) - eps (softabs) abs
============ ================================== =========================
beta may be a Python float **or** a tensor / nn.Parameter for a
learnable, differentiable temperature (see
omnibias.torch.tempered_blocks).
Numerical scope: shared coefficients and formulas do not establish universal floating-point identity across backends, devices, compiler transforms, or reduction orders. Compare the chosen path with dtype-appropriate tolerances; even float64 is not a general bitwise-equality guarantee. Algebraic initialization identities likewise need a numerical check for the selected coefficients and reduction order.
Shared formulas do not imply universal floating-point identity. Verify dtype, shape, derivative-order support, and the stated numerical assumptions for the path you use.
SOFT_RELU
omnibias.torch.activations.tempered.SOFT_RELUSource ↗SOFT_RELU = register_activation(make_soft_relu_spec())This exported constant is defined by the expression above. Its exact value and annotation come from the pinned source; use it with the module contract.
Read the implementation
SOFT_RELU = register_activation(make_soft_relu_spec())SOFT_STEP
omnibias.torch.activations.tempered.SOFT_STEPSource ↗SOFT_STEP = register_activation(make_soft_step_spec())This exported constant is defined by the expression above. Its exact value and annotation come from the pinned source; use it with the module contract.
Read the implementation
SOFT_STEP = register_activation(make_soft_step_spec())make_soft_leaky_relu_spec
omnibias.torch.activations.tempered.make_soft_leaky_relu_specSource ↗make_soft_leaky_relu_spec(negative_slope: float=0.01, beta: float | Tensor=_DEFAULT_BETA, *, name: str='soft_leaky_relu') -> ActivationSpec[Tensor]Smooth leaky ReLU alpha*z + (1-alpha)*soft_relu(z; beta) (all orders).
Recovers leaky_relu with the given negative_slope as beta -> inf.
Parameters
| Name | Type & default | Contract |
|---|---|---|
negative_slopepositional-or-keyword | floatDefault: 0.01 | See the operation contract and implementation below. |
betapositional-or-keyword | float | TensorDefault: _DEFAULT_BETA | See the operation contract and implementation below. |
namekeyword-only | strDefault: 'soft_leaky_relu' | See the operation contract and implementation below. |
ActivationSpec[Tensor]Executed regression · soft leaky relu converges
Recorded coverage confirms this test executed the API body. This example belongs to the pinned upstream test suite. It may use fixtures and imports from its full file. Run it from that checkout with its test dependencies installed.
def test_soft_leaky_relu_converges() -> None:
alpha = 0.1
leaky = get_activation("leaky_relu") # slope 0.01; build matching alpha spec
from omnibias.torch.activations.piecewise import make_leaky_relu_spec
hard = make_leaky_relu_spec(alpha)
z = torch.tensor([-3.0, -1.0, -0.3, 0.3, 1.0, 3.0], dtype=torch.float64)
err = (make_soft_leaky_relu_spec(alpha, 128.0).forward(z) - hard.forward(z)).abs().max()
assert err.item() < 1e-2
# derivative arms: alpha on z<0, 1 on z>0.
d = make_soft_leaky_relu_spec(alpha, 128.0).fastpath(z, 1)
expected = torch.where(z > 0, torch.ones_like(z), torch.full_like(z, alpha))
torch.testing.assert_close(d, expected, rtol=0, atol=1e-2)
assert leaky.name == "leaky_relu"python -m pytest "packages/omnibias-torch/tests/test_tempered.py::test_soft_leaky_relu_converges" -qRead the implementation
def make_soft_leaky_relu_spec(
negative_slope: float = 0.01,
beta: float | Tensor = _DEFAULT_BETA,
*,
name: str = "soft_leaky_relu",
) -> ActivationSpec[Tensor]:
"""Smooth leaky ReLU ``alpha*z + (1-alpha)*soft_relu(z; beta)`` (all orders).
Recovers ``leaky_relu`` with the given ``negative_slope`` as ``beta -> inf``.
"""
alpha = negative_slope
one_minus_alpha = 1.0 - alpha
def _soft_relu_nth(z: Tensor, n: int) -> Tensor:
return (beta ** (n - 1)) * softplus_nth_derivative(beta * z, n)
def fwd(z: Tensor) -> Tensor:
return alpha * z + one_minus_alpha * (F.softplus(beta * z) / beta)
def fp(z: Tensor, n: int) -> Tensor:
if n < 0:
raise ValueError(f"order n must be >= 0, got {n}.")
if n == 0:
return fwd(z)
soft = one_minus_alpha * _soft_relu_nth(z, n)
if n == 1:
return torch.full_like(z, alpha) + soft
return soft
def deriv(z: Tensor) -> Tensor:
return fp(z, 1)
return ActivationSpec(
name=name,
forward=fwd,
derivative=deriv,
fastpath=fp,
riccati_polynomial=None,
noise_model="none",
operator_role=(
"Smooth leaky ReLU: alpha*z + (1-alpha)*softplus(beta z)/beta "
"-> leaky_relu as beta -> inf."
),
)make_soft_relu_spec
omnibias.torch.activations.tempered.make_soft_relu_specSource ↗make_soft_relu_spec(beta: float | Tensor=_DEFAULT_BETA, *, name: str='soft_relu') -> ActivationSpec[Tensor]Smooth ReLU surrogate softplus(beta z) / beta (all-orders tower).
soft_relu^(n)(z) = beta**(n-1) * softplus^(n)(beta z); n = 2 is the
beta-scaled logistic bump that concentrates into relu's Dirac delta as
beta -> inf.
Parameters
| Name | Type & default | Contract |
|---|---|---|
betapositional-or-keyword | float | TensorDefault: _DEFAULT_BETA | See the operation contract and implementation below. |
namekeyword-only | strDefault: 'soft_relu' | See the operation contract and implementation below. |
ActivationSpec[Tensor]Executed regression · forward and derivative route through tower
Recorded coverage confirms this test executed the API body. This example belongs to the pinned upstream test suite. It may use fixtures and imports from its full file. Run it from that checkout with its test dependencies installed.
def test_forward_and_derivative_route_through_tower() -> None:
z = torch.linspace(-3.0, 3.0, 41, dtype=torch.float64)
for spec in (make_soft_relu_spec(2.0), make_soft_step_spec(1.5)):
torch.testing.assert_close(spec.forward(z), spec.fastpath(z, 0))
assert spec.derivative is not None
torch.testing.assert_close(spec.derivative(z), spec.fastpath(z, 1))python -m pytest "packages/omnibias-torch/tests/test_tempered.py::test_forward_and_derivative_route_through_tower" -qRegression example · swish matches autograd
This example belongs to the pinned upstream test suite. It may use fixtures and imports from its full file. Run it from that checkout with its test dependencies installed.
def test_swish_matches_autograd() -> None:
spec = make_swish_spec(1.5)
z = torch.linspace(-4.0, 4.0, 41, dtype=torch.float64)
for n in range(1, 5):
closed = spec.fastpath(z, n)
auto = _nth_autograd(spec.forward, z, n)
torch.testing.assert_close(closed, auto, rtol=1e-6, atol=1e-6, msg=f"swish n={n}")python -m pytest packages/omnibias-torch/tests/test_tempered.py -k test_swish_matches_autograd -qRead the implementation
def make_soft_relu_spec(
beta: float | Tensor = _DEFAULT_BETA, *, name: str = "soft_relu"
) -> ActivationSpec[Tensor]:
"""Smooth ReLU surrogate ``softplus(beta z) / beta`` (all-orders tower).
``soft_relu^(n)(z) = beta**(n-1) * softplus^(n)(beta z)``; ``n = 2`` is the
beta-scaled logistic bump that concentrates into ``relu``'s Dirac delta as
``beta -> inf``.
"""
return tempered(
SOFTPLUS,
beta,
scale="one_over_beta",
name=name,
operator_role=(
"Smooth ReLU: softplus(beta z)/beta -> relu as beta -> inf; "
"the n=2 bump -> delta."
),
limit_neg_inf=0.0,
)make_soft_step_spec
omnibias.torch.activations.tempered.make_soft_step_specSource ↗make_soft_step_spec(beta: float | Tensor=_DEFAULT_BETA, *, name: str='soft_step') -> ActivationSpec[Tensor]Smooth step surrogate sigmoid(beta z) (all-orders tower).
soft_step^(n)(z) = beta**n * sigma^(n)(beta z) -> Heaviside as
beta -> inf.
Parameters
| Name | Type & default | Contract |
|---|---|---|
betapositional-or-keyword | float | TensorDefault: _DEFAULT_BETA | See the operation contract and implementation below. |
namekeyword-only | strDefault: 'soft_step' | See the operation contract and implementation below. |
ActivationSpec[Tensor]Executed regression · forward and derivative route through tower
Recorded coverage confirms this test executed the API body. This example belongs to the pinned upstream test suite. It may use fixtures and imports from its full file. Run it from that checkout with its test dependencies installed.
def test_forward_and_derivative_route_through_tower() -> None:
z = torch.linspace(-3.0, 3.0, 41, dtype=torch.float64)
for spec in (make_soft_relu_spec(2.0), make_soft_step_spec(1.5)):
torch.testing.assert_close(spec.forward(z), spec.fastpath(z, 0))
assert spec.derivative is not None
torch.testing.assert_close(spec.derivative(z), spec.fastpath(z, 1))python -m pytest "packages/omnibias-torch/tests/test_tempered.py::test_forward_and_derivative_route_through_tower" -qRegression example · soft step beta scaling identity
This example belongs to the pinned upstream test suite. It may use fixtures and imports from its full file. Run it from that checkout with its test dependencies installed.
def test_soft_step_beta_scaling_identity(beta: float) -> None:
"""``soft_step^(n)(z) == beta**n * sigmoid^(n)(beta z)``."""
spec = make_soft_step_spec(beta)
sigmoid = get_activation("sigmoid")
z = torch.linspace(-3.0, 3.0, 41, dtype=torch.float64)
for n in range(0, 6):
lhs = spec.fastpath(z, n)
rhs = (beta**n) * sigmoid.fastpath(beta * z, n)
torch.testing.assert_close(lhs, rhs, msg=f"soft_step identity n={n}")python -m pytest packages/omnibias-torch/tests/test_tempered.py -k test_soft_step_beta_scaling_identity -qRead the implementation
def make_soft_step_spec(
beta: float | Tensor = _DEFAULT_BETA, *, name: str = "soft_step"
) -> ActivationSpec[Tensor]:
"""Smooth step surrogate ``sigmoid(beta z)`` (all-orders tower).
``soft_step^(n)(z) = beta**n * sigma^(n)(beta z) -> Heaviside`` as
``beta -> inf``.
"""
return tempered(
SIGMOID,
beta,
scale="unit",
name=name,
aliases=("soft_heaviside",) if name == "soft_step" else (),
operator_role=(
"Smooth Heaviside: sigmoid(beta z) -> step as beta -> inf."
),
limit_pos_inf=1.0,
limit_neg_inf=0.0,
)make_swish_spec
omnibias.torch.activations.tempered.make_swish_specSource ↗make_swish_spec(beta: float | Tensor=_DEFAULT_BETA, *, name: str='swish') -> ActivationSpec[Tensor]Swish z * sigmoid(beta z) with tunable beta (silu is beta=1).
Exact all-orders via Leibniz on z * sigmoid(beta z):
swish^(n)(z) = z * beta**n * sigma^(n)(beta z) + n * beta**(n-1) *
sigma^(n-1)(beta z).
Parameters
| Name | Type & default | Contract |
|---|---|---|
betapositional-or-keyword | float | TensorDefault: _DEFAULT_BETA | See the operation contract and implementation below. |
namekeyword-only | strDefault: 'swish' | See the operation contract and implementation below. |
ActivationSpec[Tensor]Executed regression · swish beta1 is silu
Recorded coverage confirms this test executed the API body. This example belongs to the pinned upstream test suite. It may use fixtures and imports from its full file. Run it from that checkout with its test dependencies installed.
def test_swish_beta1_is_silu() -> None:
swish = make_swish_spec(1.0)
silu = get_activation("silu")
z = torch.linspace(-4.0, 4.0, 41, dtype=torch.float64)
for n in range(0, 6):
torch.testing.assert_close(
swish.fastpath(z, n), silu.fastpath(z, n), msg=f"swish(beta=1) != silu at n={n}"
)python -m pytest "packages/omnibias-torch/tests/test_tempered.py::test_swish_beta1_is_silu" -qRegression example · swish matches autograd
This example belongs to the pinned upstream test suite. It may use fixtures and imports from its full file. Run it from that checkout with its test dependencies installed.
def test_swish_matches_autograd() -> None:
spec = make_swish_spec(1.5)
z = torch.linspace(-4.0, 4.0, 41, dtype=torch.float64)
for n in range(1, 5):
closed = spec.fastpath(z, n)
auto = _nth_autograd(spec.forward, z, n)
torch.testing.assert_close(closed, auto, rtol=1e-6, atol=1e-6, msg=f"swish n={n}")python -m pytest packages/omnibias-torch/tests/test_tempered.py -k test_swish_matches_autograd -qRead the implementation
def make_swish_spec(
beta: float | Tensor = _DEFAULT_BETA, *, name: str = "swish"
) -> ActivationSpec[Tensor]:
"""Swish ``z * sigmoid(beta z)`` with tunable ``beta`` (``silu`` is beta=1).
Exact all-orders via Leibniz on ``z * sigmoid(beta z)``:
``swish^(n)(z) = z * beta**n * sigma^(n)(beta z) + n * beta**(n-1) *
sigma^(n-1)(beta z)``.
"""
def fwd(z: Tensor) -> Tensor:
return z * torch.sigmoid(beta * z)
def fp(z: Tensor, n: int) -> Tensor:
if n < 0:
raise ValueError(f"order n must be >= 0, got {n}.")
if n == 0:
return fwd(z)
bz = beta * z
term_a = z * (beta**n) * sigmoid_nth_derivative(bz, n)
term_b = n * (beta ** (n - 1)) * sigmoid_nth_derivative(bz, n - 1)
return term_a + term_b
def deriv(z: Tensor) -> Tensor:
return fp(z, 1)
return ActivationSpec(
name=name,
forward=fwd,
derivative=deriv,
fastpath=fp,
riccati_polynomial=None,
noise_model="none",
operator_role="Swish: z * sigmoid(beta z); exact all-orders (silu is beta=1).",
)