Module activation 

Activation & normalization functions — the STEM definitions of the element-wise and reduction operations a transformer forward pass is built from.

These are not a proprietary “AI engine”; they are standard mathematics:

• activations (ReLU, sigmoid, tanh, SiLU/Swish, GELU) — element-wise nonlinear maps;
• softmax — the normalized exponential, a projection onto the probability simplex;
• RMS / layer normalization — statistical rescaling (variance / mean-and-variance).

This module is the canonical, inspectable home for that math. The LLM runtime (gguf_bridge) holds inline f32 hot-path versions and promoted GPU kernels; those are backends of these definitions and are checked against them (the same arrangement proved for GEMM in solvers::linear_algebra::gemm). gguf itself is only a weight file format — the mathematics lives here.

All functions operate in place on caller-owned f64 slices — zero allocation.

Functions

gelu

GELU (Gaussian Error Linear Unit), tanh approximation: 0.5·x·(1 + tanh(√(2/π)·(x + 0.044715·x³))). The standard GPT-2/transformer GELU.

layer_norm

Layer normalization in place: x_i ← ((x_i − μ) / sqrt(σ² + eps)) · weight_i + bias_i, with μ, σ² the mean and (population) variance over x. weight/bias match x.

relu

ReLU: max(0, x), element-wise.

rms_norm

RMS normalization in place: x_i ← (x_i / sqrt(mean(x²) + eps)) · weight_i. No mean subtraction (the Llama/transformer RMSNorm). weight must match x in length; shorter is honoured up to the common length.

sigmoid

Logistic sigmoid: σ(x) = 1 / (1 + e^{-x}), element-wise.

silu

SiLU / Swish: x · σ(x) = x / (1 + e^{-x}), element-wise (Llama/SmolLM2 gate activation).

softmax

Softmax in place: softmax(x)_i = e^{x_i} / Σ_j e^{x_j}, computed in the numerically stable shifted form e^{x_i − max} / Σ e^{x_j − max}. After the call x sums to 1 (a probability distribution). A length-0 slice is left unchanged.

tanh

Hyperbolic tangent, element-wise.