Choosing an activation

The base activation determines what your operator-typed layer can compute. Pick it with these four questions.

The decision tree

1. What derivative orders do you need?

n ≥ 2 (Laplacian, Hessian, biharmonic, …): you must use the smooth (Riccati) family — tanh, sigmoid, softplus, gaussian. Only these have a closed-form tower at every order.
n ≤ 1 (value or gradient only): the classical or proximal families are fine.

Order limits are real

relu, silu, and gelu only support op="identity" and op="grad". Requesting op="laplacian" on them raises a clear error. The exact per-order limits are in the Stability matrix.

2. What operator role do you need?

If you want the K=2 collapse to be a classical operator, match the role:

You want	Use	Collapse output
L1 proximal / soft-shrink (LASSO ISTA)	`huber`	`clip(z, -τ, τ)`
Cauchy IRLS weight	`arctan`	`1/(1 + z²)`
Redescending M-estimator	`log1pu2`	`2z/(1 + z²)`

3. What noise model does your loss assume?

If σ is the log-partition of a GLM family, matching it aligns your layer with the upstream likelihood. The noise_model field on each ActivationSpec records this.

4. What inductive bias do you want at init?

Lemma-1 initialization makes the layer behave as the base σ at step zero, no matter what K is — so you can adopt an operator role without disturbing the network's initial function.

Practical defaults

PINNs / high-order PDEs: start with tanh; try gaussian when the solution is bump-like or when Hermite structure helps.
VMC envelopes: follow the system; the FermiNet bridge handles the envelope derivatives.
Robust regression layers: huber or arctan per the role table.
Reusing a pretrained backbone: keep its relu/gelu/silu and use op="identity".

The decision tree​

1. What derivative orders do you need?​

2. What operator role do you need?​

3. What noise model does your loss assume?​

4. What inductive bias do you want at init?​

Practical defaults​

See also​