Skip to main content

swiglu_ffn

Function swiglu_ffn 

Source
pub fn swiglu_ffn(
    d_model: usize,
    d_ff: usize,
    x: &[f64],
    w_gate: &[f64],
    w_up: &[f64],
    w_down: &[f64],
    gate_buf: &mut [f64],
    up_buf: &mut [f64],
    out: &mut [f64],
) -> Result<(), SolversError>
Expand description

Compute out = W_down · ( SiLU(W_gate · x) ⊙ (W_up · x) ), row-major, caller-owned.

  • x: input, length d_model
  • w_gate, w_up: d_ff × d_model each
  • w_down: d_model × d_ff
  • gate_buf, up_buf: scratch, length d_ff each (overwritten)
  • out: result, length d_model (overwritten)

Fails closed (SolversError::InvalidDimension) on any shape mismatch.