Skip to main content

VOCAB_CHUNK_ROWS

Constant VOCAB_CHUNK_ROWS 

Source
pub const VOCAB_CHUNK_ROWS: usize = MAX_STACK_GEMM_OUT; // 10_240usize
Expand description

Vocabulary projection rows per chunked logits sweep. 10240 is the native GEMM output-buffer ceiling and a 256-row multiple, so resident logits chunk offsets stay storage-binding aligned while the current 49k-vocab model drops from six output chunks/token to five.