ADR 0010: Prompt-lookup speculative decode is default-ON, with a runtime mode switch

Status

Accepted (2026-07-05, 0.0.28) — directed by Timothy. Ships the W6a prompt-lookup speculative decoder on by default, accepting rare benign near-tie divergence from single-token decode, with an explicit mode switch so either mode can be selected at launch or at runtime (incl. from the desktop UI).

Context

W6a added prompt-lookup (LLMA / n-gram) speculative decoding to the native decode loop (inference_agent.rs), backed by a batched verify forward (gguf_bridge/verify_arena.rs). Per step, when no sieve/sampler/sparse-route is active, it drafts the next few tokens from an earlier recurrence in the context, verifies the whole draft in one batched forward, and emits the longest greedily-agreeing prefix plus the model’s own correction token. On this compute-bound class of GPU it is the first real steady-state decode-tok/s win in the optimization programme (fence-latency wins like W1/W3 are latent here): measured ~3–12× decode tok/s on repetitive / quoting / structured / code text; on novel prose the proposer drafts little and costs ≈nothing.

The open question was transparency: is speculative output identical to ordinary single-token decode? Verify selects with a full-logit CPU argmax over its batched forward; the default decode path (resident_decode) selects with a GPU top-1 block reduction over its single-token forward. A diagnostic (a6 extended with a resident-path reference, spec_verify_probe_blocking) established:

Achieving bit-perfect transparency would require making verify’s batched forward bit-identical to the resident single-token forward at the ULP level — unifying two different orchestrations’ float-reduction order — a large, deep change for effectively no user-visible benefit (the only positions that would change are ones where the model itself is indifferent).

The exact-output gate a6a confirms the wiring is correct: against a consistent selection method (CPU argmax on both sides), speculative decode is bit-identical to greedy, with 48/48 draft tokens accepted on a repetitive prompt.

Decision

Ship prompt-lookup speculative decode ON by default, and expose an explicit mode switch.

  1. QUALIA_LLM_SPEC_DECODE defaults ON (inference_bench.rs). Speculative decode runs whenever no sieve, sampler, or sparse-attention route is active and the full model runs. Ineligible steps fall back to ordinary single-token decode unchanged.
  2. We accept the rare, benign near-tie divergence from single-token decode. It is confined to positions where the model is genuinely ambivalent (both tokens equally valid) and is the same class as the already-tolerated a1a near-tie. We do not pursue bit-perfect transparency (poor return).
  3. Mode switch — three equivalent controls, so either mode is always selectable, including from the desktop UI / host:
    • Launch: env QUALIA_LLM_SPEC_DECODE=0 (single-token) / =1 (speculative). The env var, when set, overrides the runtime flag in both directions. Read by the native engine (desktop + daemon).
    • Runtime: qualia_core_db::llm_bench::set_spec_decode(bool) — the host/UI calls this to flip modes live between inferences.
    • Read-back: spec_decode_enabled() returns the effective mode (for reflecting state in a UI). The webizen-desktop UI wires a control to set_spec_decode / the env var (that wiring lives in the webizen-browser repo). Speculative decode is native-only — the decode-loop branch is #[cfg(not(target_arch = "wasm32"))], so the wasm build always runs single-token decode and the switch is a no-op there.

Consequences

References