Activation dictionary
An activation specification supplies the value function and the derivative paths supported by that backend. The public registry is the source of supported names; custom factories create parameterized specifications.
| Family | Mathematical behavior | What to check |
|---|---|---|
| Sigmoid and tanh | Polynomial recurrences in the activation value | Floating-point growth and cancellation at high order |
| Gaussian | Hermite recurrence | The source's normalization and width convention |
| Softplus, SiLU and GELU | Dedicated smooth derivative paths | Supported path in the selected backend and release |
| Piecewise activations | Almost-everywhere or regular-part derivatives | Values at knots; omitted distributional impulses |
| Huber and other proximal constructions | Clipping, robust weighting, or smooth surrogates | Threshold domain and per-order support |
| Tempered activations | A parameter controls transition sharpness | Temperature scaling of derivatives and gradient saturation |
A function being smooth does not imply every implementation provides every derivative order. Conversely, piecewise functions may expose higher regular-part derivatives without representing the impulses at their knots. Do not interpret those values as a globally smooth derivative tower.
The published classical Torch implementation includes all-order SiLU and GELU paths; the older restriction to orders zero and one is obsolete. arctan and log1pu2 currently provide orders zero, one, and two in the Torch proximal implementation. Read the backend-specific reference before changing a PDE's activation.
The derivative of the Huber loss is clipping. The proximal operator of the L1 norm is soft-thresholding; those are different operations. Use the documented operator expression, rather than treating an informal “proximal” family label as an equivalence.
Start with the tested first derivative, then inspect Torch activations, JAX activations, or Keras activations.