Skip to main content

VOCAB_CHUNK_ROWS

Constant VOCAB_CHUNK_ROWS 

Source
pub const VOCAB_CHUNK_ROWS: usize = 10240;
Expand description

Vocabulary projection rows per chunked logits sweep. 10240 is the native GEMM output-buffer ceiling and a 256-row multiple, so resident logits chunk offsets stay storage-binding aligned while the current 49k-vocab model drops from six output chunks/token to five.