Choosing an activation

Choose an activation for the operator you will actually evaluate. A fourth-order residual needs a usable fourth derivative throughout the training domain; a binary gate needs a controllable transition and meaningful parameter gradients.

For smooth PDE residuals

Start with the tested tanh path in the high-order residual tutorial. Confirm the required order, dtype, and architecture path in the API contract. Compare a small batch with an independent formula or automatic-differentiation reference before scaling up. A closed-form expression still incurs floating-point error and can be poorly scaled.

For localized features

Gaussian and related bases can express localized structure. Check the normalization, width parameter, and derivative convention in the selected specification. Rescaling the input by a factor changes an order-n spatial derivative by the corresponding n-th power, so width and coordinate normalization belong in your conditioning analysis.

For hardening decisions

Use a smooth finite-temperature gate while learning regional boundaries. As inverse temperature increases, gradients concentrate near a boundary and may vanish away from it. The hard limit is an inference choice, not a guarantee that training remains well conditioned. See soft regional models.

For existing neural networks

Preserve the network's activation definition, including approximation settings. The current classical implementations include more than first derivatives, but support must be checked per backend. Piecewise derivatives use explicit conventions at knots and do not include distributional impulses unless the API says so.

Before training

Record the activation name, custom factory parameters, backend version, floating-point dtype, input scale, and maximum derivative order. Exercise the exact path that feeds your loss and assert that parameter gradients remain finite. Use the activation dictionary and API explorer to find the matching contract.