Module hyper_connection
Expand description
Qwen4Exp four-stream Hyper-Connection vector operations.
Projection weights are streamed elsewhere. This module applies their per-stream gates to hidden-state buffers supplied by the caller.
Functions§
- group_
rms_ norm_ in_ place - In-place variant for a transient stream buffer whose unnormalized values
are not needed by the caller after the operation. Same pre-folded gamma
semantics as
group_rms_norm_into. - group_
rms_ norm_ into - Normalize each Hyper-Connection stream independently with RMSNorm.
The GGUF converter pre-folds the zero-centered
(1 + weight)scale into the stored gamma, so the checkpoint bytes multiply directly — callers must not add a further bias (llama.cppbuild_hc_mix). - inject_
stream_ update - Add a block update into each Hyper stream.
inject_gateis the sigmoid output of the streamed HC inject projection; Qwen scales it by two. - mix_
streams_ into - Form the token-mixer or MoE input from the gated average of Hyper streams.
mix_gateis the sigmoid output of the streamed HC up-projection.