Skip to main content

mix_streams_into

Function mix_streams_into 

pub fn mix_streams_into(
    normalized: &[f32],
    mix_gate: &[f32],
    stream_count: usize,
    hidden: usize,
    out: &mut [f32],
) -> Result<(), QwenNumericError>
Expand description

Form the token-mixer or MoE input from the gated average of Hyper streams. mix_gate is the sigmoid output of the streamed HC up-projection.