pub struct TokenTrie { /* private fields */ }Expand description
A prefix trie over token byte strings. Built once from the vocabulary.
Implementations§
Source§impl TokenTrie
impl TokenTrie
Sourcepub fn build(vocab: &[(u32, &[u8])]) -> Self
pub fn build(vocab: &[(u32, &[u8])]) -> Self
Build a trie from a vocabulary: (token_id, token_bytes) pairs.
Cold path — allocation is expected here.
Sourcepub fn build_from_strings(vocab: &[(u32, String)]) -> Self
pub fn build_from_strings(vocab: &[(u32, String)]) -> Self
Build from a tokenizer that provides token_id → string lookups.
Sourcepub fn collect_valid(
&self,
byte_allowed: &dyn Fn(u8, usize) -> bool,
out: &mut [u32],
) -> usize
pub fn collect_valid( &self, byte_allowed: &dyn Fn(u8, usize) -> bool, out: &mut [u32], ) -> usize
Walk the trie from the root, collecting all token IDs reachable via
bytes that pass the byte_allowed predicate. This is the core mask
computation: a token is valid iff every byte in its string passes
the predicate at the corresponding depth.
Hot path — uses caller-supplied output buffer (zero-heap).
Sourcepub fn vocab_size(&self) -> usize
pub fn vocab_size(&self) -> usize
Number of tokens in the vocabulary.
Trait Implementations§
Auto Trait Implementations§
impl Freeze for TokenTrie
impl RefUnwindSafe for TokenTrie
impl Send for TokenTrie
impl Sync for TokenTrie
impl Unpin for TokenTrie
impl UnsafeUnpin for TokenTrie
impl UnwindSafe for TokenTrie
Blanket Implementations§
§impl<S, A> Aggregate<Result<S, Error>> for Awhere
A: Aggregate<S>,
impl<S, A> Aggregate<Result<S, Error>> for Awhere
A: Aggregate<S>,
Aggregate shares in an MPC protocol.
Source§impl<T> BorrowMut<T> for Twhere
T: ?Sized,
impl<T> BorrowMut<T> for Twhere
T: ?Sized,
Source§fn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
Mutably borrows from an owned value. Read more
impl<ST, DT> CastableFrom<ST, Initialized, Initialized> for DT
impl<ST, DT> CastableFrom<ST, Uninit, Uninit> for DT
Source§impl<T> CloneToUninit for Twhere
T: Clone,
impl<T> CloneToUninit for Twhere
T: Clone,
§impl<T> Instrument for T
impl<T> Instrument for T
§fn instrument(self, span: Span) -> Instrumented<Self> ⓘ
fn instrument(self, span: Span) -> Instrumented<Self> ⓘ
Source§impl<T> IntoEither for T
impl<T> IntoEither for T
Source§fn into_either(self, into_left: bool) -> Either<Self, Self> ⓘ
fn into_either(self, into_left: bool) -> Either<Self, Self> ⓘ
Converts
self into a Left variant of Either<Self, Self>
if into_left is true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read moreSource§fn into_either_with<F>(self, into_left: F) -> Either<Self, Self> ⓘ
fn into_either_with<F>(self, into_left: F) -> Either<Self, Self> ⓘ
Converts
self into a Left variant of Either<Self, Self>
if into_left(&self) returns true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read more