Skip to main content

Module speculative_decode

Module speculative_decode 

Source
Expand description

A2 — Speculative Constrained Decoding (DOMINO).

Subword-aligned prefix-trie token masking integrated into the in-process QTensorEngine decode loop. The approach:

  1. TokenTrie — a prefix trie over token byte strings, built once from the tokenizer vocabulary (cold path).
  2. GrammarState — a state machine tracking which character classes and keywords are valid at the current parse position, based on the VibeScript 0.1 GBNF grammar.
  3. DominoMasker — combines the trie and grammar state to produce a logit mask, setting disallowed tokens to -inf before sampling.

The mask is applied per-token on the hot path using caller-supplied buffers (zero-heap). The trie is walked incrementally: at each decode step, the grammar state determines valid byte ranges, and the trie is traversed to find all token IDs whose byte sequences are valid continuations.

§Integration

The masker is called between logit readback and the sampler:

logits = engine.readback_logits();
masker.apply_mask(&mut logits, &accepted_tokens);
token = sampler.sample(&mut logits, &ctx);

When no grammar is active (unconstrained mode), apply_mask is a no-op.

Structs§

DominoMasker
The DOMINO constrained decoding masker. Combines a token trie with a grammar state machine to produce logit masks.
DominoPreparationBudget
GrammarStateMachine
The grammar state machine. Tracks the current parse position and a stack of nested contexts (blocks, parens, triples, etc.) for proper nesting.
PreparedDomino
TokenTrie
A prefix trie over token byte strings. Built once from the vocabulary.

Enums§

GrammarState
VibeScript 0.1 grammar parse states. These correspond to the non-terminals in vibe-0.1.gbnf that determine which characters are valid next.

Constants§

MAX_TRIE_DEPTH
Maximum trie depth (longest token in bytes we support).