Activation identities and bias collapse

A supported activation derivative can be evaluated using an algebraic identity. For tanh, write t = tanh(z) and start with T₀(t)=t. The recurrence is:

Tₙ₊₁(t) = (1 − t²) Tₙ′(t)
tanh⁽ⁿ⁾(z) = Tₙ(tanh(z))

One base tanh evaluation can be reused across derivative orders. The arithmetic required to evaluate the polynomial still grows with its degree. Floating-point cancellation remains possible; closed form does not imply universal machine-precision accuracy.

The bias-collapse limit

A properly weighted and normalized group of shifted activations approaches a derivative:

lim δ→0  δ⁻ⁿ Σⱼ₌₀ⁿ (−1)ⁿ⁻ʲ binomial(n,j) σ(z+jδ) = σ⁽ⁿ⁾(z)

The coefficients and normalization are essential. An unweighted sum of coalescing activations is not an arbitrary derivative. The runtime kernel evaluates the analytic limit directly, avoiding the subtraction of nearby samples used in finite differences.

Derivatives through a network

Activation identities do not alone give every network derivative. Taylor jets compose these identities through supported layers. Direct Laplacian contractions compute a particular quantity without constructing the full derivative tensor.

A different limit: temperature

sigmoid(beta*z) approaches a hard gate as beta increases, away from ties. This hardens a decision; it does not extract an activation derivative. Regional models use smooth gates at finite inverse temperature to preserve training gradients.

Run a pure-Python coefficient example or evaluate a derivative in PyTorch.