Struct GgufTokenizer 

pub struct GgufTokenizer {
    pub vocab: Vec<String>,
    pub bos_token_id: u32,
    pub eos_token_id: u32,
    pub add_bos_token: bool,
    pub pre_type: String,
    /* private fields */
}

Vocabulary and BOS/EOS metadata extracted from a GGUF KV section. Used by infer_local_model() to encode prompts and decode output token IDs.

Fields

vocab: Vec<String>

Token ID → string (index = token ID).

bos_token_id: u32

eos_token_id: u32

add_bos_token: bool

tokenizer.ggml.add_bos_token — prepend BOS before prompt tokens when true.

pre_type: String

tokenizer.ggml.pre — e.g. smollm, gpt2; drives pretokenization.

Implementations

impl GgufTokenizer

pub fn decode_token_bytes_into(&self, id: u32, out: &mut [u8]) -> Option<usize>

Decode one token into caller-owned storage.

Returns None for an unknown token or when out is too small. A token’s decoded byte representation is never longer than its vocabulary string, so callers can use a bounded stack buffer without intermediate heap allocation.

impl GgufTokenizer

pub fn from_gguf(mmap: &[u8]) -> Self

Parse vocab + BOS/EOS from a memory-mapped GGUF v2/v3 file. Falls back to Default (byte-level) on any parse error.

pub fn to_p64_section(&self) -> Vec<u8>

Phase 4 v3 / v2: serialize the tokenizer into a compact, contiguous P64 section (no page alignment needed). Fields: vocab / merges / bos / eos / add_bos / pre, plus (v2) the stop-token set so decode does not re-guess chat ends. Derived maps are rebuilt by [from_p64_section]. Heap use here is load-time only.

pub fn to_q42_section(&self) -> Vec<u8>

👎Deprecated:

use to_p64_section

Compatibility alias for the historical pre-P64 method name.

pub fn from_p64_section(data: &[u8]) -> Option<Self>

Phase 4 v3: rebuild a tokenizer from a P64 tokenizer section — bypasses GGUF KV string-key parsing entirely. Fully bounds-checked (the section is untrusted input). Returns None on any malformed field.

pub fn rebuild_stop_token_ids(&mut self)

Rebuild the decode stop set from eos_token_id + known chat-end specials in vocab. Call after any mutation of eos / token_to_id_map (load paths do this automatically).

pub fn is_stop_token(&self, id: u32) -> bool

Whether id is a generation stop token (eos and/or chat end-of-turn).

pub fn stop_tokens(&self) -> &[u32]

Slice of active stop-token ids (for logging / q42 export).

pub fn merge_stop_token_ids(&mut self, extra: &[u32])

Merge extra stop ids (e.g. from a model’s canonical .q42 metadata) into the stop set. Does not allocate; drops overflow past MAX_STOP_TOKEN_IDS.

pub fn encode_prompt(&self, text: &str) -> Vec<u32>

Tokenize text, prepending [bos_token_id] when [add_bos_token] is set and absent.

pub fn chat_family(&self) -> ChatFamily

Detect this model’s chat-template family from the special tokens present in its vocab.

pub fn apply_chat_template(&self, system: Option<&str>, user: &str) -> String

Wrap a user prompt (and optional system message) in the model’s chat template, cueing the assistant turn so an instruct model answers instead of degenerating. The tokenizer BOS is still prepended by [encode_prompt]; it is NOT embedded here (avoids a double BOS). Returns the raw prompt unchanged when no chat family is recognised.

pub fn encode_chat_prompt(&self, user: &str) -> Vec<u32>

Apply the model’s chat template (if any), then tokenize (+BOS per add_bos_token). This is the path for interactive chat/instruct inference; [encode_prompt] stays the raw-completion path. Chat models without a recognised family fall back to the raw prompt.

pub fn format_ids_for_log(ids: &[u32]) -> String

Format token IDs for diagnostic logging (MC3f).

pub fn encode(&self, text: &str) -> Vec<u32>

Greedy longest-match tokenisation; falls back to single-byte encoding.

pub fn pretokenize_into( &self, text: &str, out: &mut [PretokenSpan], ) -> Result<usize, PretokenError>

Regex-free llama.cpp LLAMA_VOCAB_PRE_TYPE_SMOLLM-compatible borrowed-span split.

pub fn decode(&self, ids: &[u32]) -> String

Map token IDs → strings, joining without separator. Converts SentencePiece ▁ and GPT-2 BPE Ġ → space; <0x##> → raw byte.

pub fn decode_token_bytes_cold(&self, id: u32) -> Vec<u8>

Decode one token into the exact byte piece used by comparator APIs.

This allocates and is intended for cold diagnostics, receipts, and corpus comparison, never for the token-forward hot path.

pub fn vocab_len(&self) -> u32

pub fn merge_count(&self) -> usize

Number of BPE merges loaded from GGUF (diagnostic).

Trait Implementations

impl Default for GgufTokenizer

fn default() -> Self

256-entry byte-level fallback tokenizer — used when no GGUF is loaded.