Skip to main content

Module hyper_connection

Module hyper_connection 

Expand description

Qwen4Exp four-stream Hyper-Connection vector operations.

Projection weights are streamed elsewhere. This module applies their per-stream gates to hidden-state buffers supplied by the caller.

Functions§

group_rms_norm_in_place
In-place variant for a transient stream buffer whose unnormalized values are not needed by the caller after the operation. Same pre-folded gamma semantics as group_rms_norm_into.
group_rms_norm_into
Normalize each Hyper-Connection stream independently with RMSNorm. The GGUF converter pre-folds the zero-centered (1 + weight) scale into the stored gamma, so the checkpoint bytes multiply directly — callers must not add a further bias (llama.cpp build_hc_mix).
inject_stream_update
Add a block update into each Hyper stream. inject_gate is the sigmoid output of the streamed HC inject projection; Qwen scales it by two.
mix_streams_into
Form the token-mixer or MoE input from the gated average of Hyper streams. mix_gate is the sigmoid output of the streamed HC up-projection.