Skip to main content

Module qwen_projection

Module qwen_projection 

Expand description

Caller-buffered Qwen projection ordering and per-head numerical transforms.

Structs§

QwenHeadLayout

Enums§

QwenProjectionPacking

Functions§

gated_rms_norm
One head: RMS(input)stored_weightSiLU(z), with no zero-centered weight offset. Exact nonempty slices and positive finite epsilon; every error is atomic.
normalize_qwen_qk
L2 normalize global Q/K with sqrt(sum(x*x)+1e-6), repeat each key head Hv/Hk consecutive times, and scale Q by K^-0.5. Outputs have [Hv,K] shape. Every error leaves both output buffers unchanged.
qwen_gates
Produce explicit per-value-head decay=exp(-exp(A_log)*softplus(a+dt_bias)) and beta=sigmoid(b). Stable gates; any admission/arithmetic error is atomic.
silu_into
Stable SiLU over one token’s channels; errors leave output unchanged.
unpack_qwen_projections
Unpack one token into global Q/K/V channels and head-major Z/B/A. Exact extents, finite inputs, and packing are required before any write.