Module qwen_projection
Expand description
Caller-buffered Qwen projection ordering and per-head numerical transforms.
Structs§
Enums§
Functions§
- gated_
rms_ norm - One head: RMS(input)stored_weightSiLU(z), with no zero-centered weight offset. Exact nonempty slices and positive finite epsilon; every error is atomic.
- normalize_
qwen_ qk - L2 normalize global Q/K with sqrt(sum(x*x)+1e-6), repeat each key head Hv/Hk consecutive times, and scale Q by K^-0.5. Outputs have [Hv,K] shape. Every error leaves both output buffers unchanged.
- qwen_
gates - Produce explicit per-value-head decay=exp(-exp(A_log)*softplus(a+dt_bias)) and beta=sigmoid(b). Stable gates; any admission/arithmetic error is atomic.
- silu_
into - Stable SiLU over one token’s channels; errors leave output unchanged.
- unpack_
qwen_ projections - Unpack one token into global Q/K/V channels and head-major Z/B/A. Exact extents, finite inputs, and packing are required before any write.