Expand description
W2 — exact CPU sampling chain for decode.
The native decode loop selects tokens by pure greedy argmax, which is prone to the repetition collapse documented across the bench probes. This module adds a full, llama.cpp-compatible sampling chain that runs on the CPU over the full logit vector (read back once per token — acceptable at the W1 one-fence-per-token cost):
repetition/frequency/presence penalties → temperature → top-k → top-p → seeded draw
Design invariants:
- Greedy is a hard short-circuit.
temperature <= 0returns the argmax BEFORE any penalty or filter is applied, so greedy decode is bit-identical to the pre-W2 path and the a1a/a1c/a1d guarantees are untouched. - Deterministic. The draw uses a self-contained SplitMix64 PRNG seeded from the config;
the same seed + same logits + same context reproduce the same token, on native and wasm
(no
rand, no float transcendentals in the RNG, no platform entropy). - Exact, not top-K-approximated. The chain runs over the whole vocabulary; top-k / top-p are applied as masks, never as a lossy pre-reduction on the GPU.
- Pure + zero-GPU. Everything here is testable without a device or a model.
Structs§
- Sampler
Config - Sampling parameters.
temperature <= 0.0⇒ greedy argmax (all other fields ignored). - Sampler
State - Stateful sampler: owns the PRNG stream so successive tokens advance it deterministically.