Skip to main content

step_gated_deltanet

Function step_gated_deltanet 

pub fn step_gated_deltanet(
    state: &mut [f32],
    q: &[f32],
    k: &[f32],
    v: &[f32],
    alpha: &[f32],
    beta: &[f32],
    d_state: usize,
    d_head: usize,
    out: &mut [f32],
) -> Result<(), QwenStateError>
Expand description

Execute one step of the GatedDeltaNet recurrent state update and projection.

Mathematical formulation:

  1. Retrieve query projection: $v_{\text{pred}} = S_{t-1} \cdot k_t$
  2. Compute error / innovation: $e_t = v_t - v_{\text{pred}}$
  3. Update state with decay $\alpha$ and learning rate $\beta$: $S_t = S_{t-1} \odot \alpha_t + \beta_t (e_t \otimes k_t)$
  4. Output projection: $y_t = S_t \cdot q_t$

state: Persistent matrix of dimensions d_state * d_head. q: Query vector of length d_state. k: Key vector of length d_state. v: Value vector of length d_head. alpha: Decay factor per head (length d_head or 1). beta: Update gate per head (length d_head or 1). out: Output vector of length d_head.