Skip to main content

GgufTokenizer

Struct GgufTokenizer 

Source
pub struct GgufTokenizer {
    pub vocab: Vec<String>,
    pub bos_token_id: u32,
    pub eos_token_id: u32,
    pub add_bos_token: bool,
    pub pre_type: String,
    /* private fields */
}
Expand description

Vocabulary and BOS/EOS metadata extracted from a GGUF KV section. Used by infer_local_model() to encode prompts and decode output token IDs.

Fields§

§vocab: Vec<String>

Token ID → string (index = token ID).

§bos_token_id: u32§eos_token_id: u32§add_bos_token: bool

tokenizer.ggml.add_bos_token — prepend BOS before prompt tokens when true.

§pre_type: String

tokenizer.ggml.pre — e.g. smollm, gpt2; drives pretokenization.

Implementations§

Source§

impl GgufTokenizer

Source

pub fn decode_token_bytes_into(&self, id: u32, out: &mut [u8]) -> Option<usize>

Decode one token into caller-owned storage.

Returns None for an unknown token or when out is too small. A token’s decoded byte representation is never longer than its vocabulary string, so callers can use a bounded stack buffer without intermediate heap allocation.

Source§

impl GgufTokenizer

Source

pub fn from_gguf(mmap: &[u8]) -> Self

Parse vocab + BOS/EOS from a memory-mapped GGUF v2/v3 file. Falls back to Default (byte-level) on any parse error.

Source

pub fn from_raw_components( vocab: Vec<String>, merges_raw: Option<&[String]>, bos_id: Option<u32>, eos_id: Option<u32>, add_bos: Option<bool>, pre_type: Option<String>, ) -> Self

Construct tokenizer from parsed vocabulary components.

Source

pub fn from_hf_json(json_str: &str) -> Option<Self>

Parse Hugging Face tokenizer.json format into a GgufTokenizer.

Source

pub fn to_p64_section(&self) -> Vec<u8> ⓘ

Phase 4 v3 / v2: serialize the tokenizer into a compact, contiguous P64 section (no page alignment needed). Fields: vocab / merges / bos / eos / add_bos / pre, plus (v2) the stop-token set so decode does not re-guess chat ends. Derived maps are rebuilt by from_p64_section(). Heap use here is load-time only.

Source

pub fn to_q42_section(&self) -> Vec<u8> ⓘ

👎Deprecated:

use to_p64_section

Compatibility alias for the historical pre-P64 method name.

Source

pub fn from_p64_section(data: &[u8]) -> Option<Self>

Phase 4 v3: rebuild a tokenizer from a P64 tokenizer section — bypasses GGUF KV string-key parsing entirely. Fully bounds-checked (the section is untrusted input). Returns None on any malformed field.

Source

pub fn rebuild_stop_token_ids(&mut self)

Rebuild the decode stop set from eos_token_id + known chat-end specials in vocab. Call after any mutation of eos / token_to_id_map (load paths do this automatically).

Source

pub fn is_stop_token(&self, id: u32) -> bool

Whether id is a generation stop token (eos and/or chat end-of-turn).

Source

pub fn stop_tokens(&self) -> &[u32]

Slice of active stop-token ids (for logging / q42 export).

Source

pub fn merge_stop_token_ids(&mut self, extra: &[u32])

Merge extra stop ids (e.g. from a model’s canonical .q42 metadata) into the stop set. Does not allocate; drops overflow past MAX_STOP_TOKEN_IDS.

Source

pub fn encode_prompt(&self, text: &str) -> Vec<u32>

Tokenize text, prepending bos_token_id() when add_bos_token() is set and absent.

Source

pub fn chat_family(&self) -> ChatFamily

Detect this model’s chat-template family from the special tokens present in its vocab.

Source

pub fn apply_chat_template(&self, system: Option<&str>, user: &str) -> String

Wrap a user prompt (and optional system message) in the model’s chat template, cueing the assistant turn so an instruct model answers instead of degenerating. The tokenizer BOS is still prepended by encode_prompt(); it is NOT embedded here (avoids a double BOS). Returns the raw prompt unchanged when no chat family is recognised.

Source

pub fn encode_chat_prompt(&self, user: &str) -> Vec<u32>

Apply the model’s chat template (if any), then tokenize (+BOS per add_bos_token). This is the path for interactive chat/instruct inference; encode_prompt() stays the raw-completion path. Chat models without a recognised family fall back to the raw prompt.

Source

pub fn encode_chat_roles(&self, system: Option<&str>, user: &str) -> Vec<u32>

Apply the model’s chat template with explicit system and user roles, then tokenize.

Source

pub fn format_ids_for_log(ids: &[u32]) -> String

Format token IDs for diagnostic logging (MC3f).

Source

pub fn encode(&self, text: &str) -> Vec<u32>

Greedy longest-match tokenisation; falls back to single-byte encoding.

Source

pub fn pretokenize_into( &self, text: &str, out: &mut [PretokenSpan], ) -> Result<usize, PretokenError>

Regex-free llama.cpp LLAMA_VOCAB_PRE_TYPE_SMOLLM-compatible borrowed-span split.

Source

pub fn decode(&self, ids: &[u32]) -> String

Map token IDs → strings, joining without separator. Converts SentencePiece ▁ and GPT-2 BPE Ġ → space; <0x##> → raw byte.

Source

pub fn decode_token_bytes_cold(&self, id: u32) -> Vec<u8> ⓘ

Decode one token into the exact byte piece used by comparator APIs.

This allocates and is intended for cold diagnostics, receipts, and corpus comparison, never for the token-forward hot path.

Source

pub fn vocab_len(&self) -> u32

vocab length.

Source

pub fn merge_count(&self) -> usize

Number of BPE merges loaded from GGUF (diagnostic).

Trait Implementations§

Source§

impl Default for GgufTokenizer

Source§

fn default() -> Self

256-entry byte-level fallback tokenizer — used when no GGUF is loaded.

Auto Trait Implementations§

Blanket Implementations§

§

impl<S, A> Aggregate<Result<S, Error>> for A
where A: Aggregate<S>,

§

fn from_shares<T>(iter: T) -> Result<A, Error>
where T: IntoIterator<Item = Result<S, Error>>,

Aggregate shares in an MPC protocol.
Source§

impl<T> Any for T
where T: 'static + ?Sized,

Source§

fn type_id(&self) -> TypeId

Gets the TypeId of self. Read more
Source§

impl<T> Borrow<T> for T
where T: ?Sized,

Source§

fn borrow(&self) -> &T

Immutably borrows from an owned value. Read more
Source§

impl<T> BorrowMut<T> for T
where T: ?Sized,

Source§

fn borrow_mut(&mut self) -> &mut T

Mutably borrows from an owned value. Read more
§

impl<ST, DT> CastableFrom<ST, Initialized, Initialized> for DT
where ST: ?Sized, DT: ?Sized,

§

impl<ST, DT> CastableFrom<ST, Uninit, Uninit> for DT
where ST: ?Sized, DT: ?Sized,

§

impl<T> Downcast<T> for T

§

fn downcast(&self) -> &T

Source§

impl<T> From<T> for T

Source§

fn from(t: T) -> T

Returns the argument unchanged.

§

impl<T> Instrument for T

§

fn instrument(self, span: Span) -> Instrumented<Self> ⓘ

Instruments this type with the provided [Span], returning an Instrumented wrapper. Read more
§

fn in_current_span(self) -> Instrumented<Self> ⓘ

Instruments this type with the current Span, returning an Instrumented wrapper. Read more
Source§

impl<T, U> Into<U> for T
where U: From<T>,

Source§

fn into(self) -> U

Calls U::from(self).

That is, this conversion is whatever the implementation of From<T> for U chooses to do.

Source§

impl<T> IntoEither for T

Source§

fn into_either(self, into_left: bool) -> Either<Self, Self> ⓘ

Converts self into a Left variant of Either<Self, Self> if into_left is true. Converts self into a Right variant of Either<Self, Self> otherwise. Read more
Source§

fn into_either_with<F>(self, into_left: F) -> Either<Self, Self> ⓘ
where F: FnOnce(&Self) -> bool,

Converts self into a Left variant of Either<Self, Self> if into_left(&self) returns true. Converts self into a Right variant of Either<Self, Self> otherwise. Read more
§

impl<T> NoneValue for T
where T: Default,

§

type NoneType = T

§

fn null_value() -> T

The none-equivalent value.
§

impl<T> Pointable for T

§

const ALIGN: usize

The alignment of pointer.
§

type Init = T

The type for initializers.
§

unsafe fn init(init: <T as Pointable>::Init) -> usize

Initializes a with the given initializer. Read more
§

unsafe fn deref<'a>(ptr: usize) -> &'a T

Dereferences the given pointer. Read more
§

unsafe fn deref_mut<'a>(ptr: usize) -> &'a mut T

Mutably dereferences the given pointer. Read more
§

unsafe fn drop(ptr: usize)

Drops the object pointed to by the given pointer. Read more
§

impl<T> PolicyExt for T
where T: ?Sized,

§

fn and<P, B, E>(self, other: P) -> And<T, P>
where T: Sized + Policy<B, E>, P: Policy<B, E>,

Create a new Policy that returns [Action::Follow] only if self and other return Action::Follow. Read more
§

fn or<P, B, E>(self, other: P) -> Or<T, P>
where T: Sized + Policy<B, E>, P: Policy<B, E>,

Create a new Policy that returns [Action::Follow] if either self or other returns Action::Follow. Read more
§

impl<T> Read<Exclusive, BecauseExclusive> for T
where T: ?Sized,

Source§

impl<T> Same for T

Source§

type Output = T

Should always be Self
Source§

impl<T, U> TryFrom<U> for T
where U: Into<T>,

Source§

type Error = !

The type returned in the event of a conversion error.
Source§

fn try_from(value: U) -> Result<T, !>

Performs the conversion.
Source§

impl<T, U> TryInto<U> for T
where U: TryFrom<T>,

Source§

type Error = <U as TryFrom<T>>::Error

The type returned in the event of a conversion error.
Source§

fn try_into(self) -> Result<U, <U as TryFrom<T>>::Error>

Performs the conversion.
§

impl<T> Upcast<T> for T

§

fn upcast(&self) -> Option<&T>

§

impl<V, T> VZip<V> for T
where V: MultiLane<T>,

§

fn vzip(self) -> V

§

impl<T> WasmNotSend for T
where T: Send,

§

impl<T> WasmNotSendSync for T
where T: WasmNotSend + WasmNotSync,

§

impl<T> WasmNotSync for T
where T: Sync,

§

impl<T> WithSubscriber for T

§

fn with_subscriber<S>(self, subscriber: S) -> WithDispatch<Self> ⓘ
where S: Into<Dispatch>,

Attaches the provided Subscriber to this type, returning a [WithDispatch] wrapper. Read more
§

fn with_current_subscriber(self) -> WithDispatch<Self> ⓘ

Attaches the current default Subscriber to this type, returning a [WithDispatch] wrapper. Read more