Module inference_bench

Expand description

A0 — native LLM benchmark harness (STELLAR §A; decisions D17 + D22).

This is the shared measurement surface for the performance push. It drives the real inference path ([LocalLlmAgent::infer_local_model_streaming]) and reads per-phase timing recorded inside that same path — so the existing F16/Q8 path and the future ternary/top-k paths are measured by one harness rather than a forked benchmark loop (D22 “shared-improvement” rule). A speedup that shows up here is a real, attributable, end-to-end number, not a kernel microbenchmark.

What it reports (per model / weight policy):

cold TTFT — model not resident: wall-clock from call to first token (bundles mmap load + pipeline create + prefill + first decode);
warm TTFT — model resident (mmap adopted, pipelines still rebuilt per call in the current architecture — that cost is intentionally included, it is what A7 will attack);
prefill / decode tok/s from the internal phase split;
the load / prefill / decode wall-clock breakdown.

Honest scope of this increment (A0.1): timings are host wall-clock. GPU timestamp-query kernel isolation (D17) — requesting TIMESTAMP_QUERY on the shared device and wrapping passes with timestamp_writes — is the A0.2 follow-on; BenchResult::gpu_timestamp_supported is false until then.

Native-only: the WASM decode path is a different beast and is benchmarked in the browser harness.

Library-ized (CLAUDE.md §11) from a single inference_bench.rs into cohesive submodules — pure code motion, no behaviour change. The public surface is re-exported unchanged, so crate::inference::inference_bench::<Item> resolves exactly as before.

Re-exports§

pub use raw_decode::*;

Modules§

raw_decode: Fixed-step raw decoder benchmark.

Structs§

BenchConfig: One benchmark case: a model + prompt to drive through the real path.
BenchGpuMeta
BenchResult: A single benchmark row — JSON/CSV serializable.
LlmPhaseSnapshot: Snapshot of the phase counters (last completed inference).
ModelMeta: Model metadata captured at residency mount (best-effort).

Enums§

GemmBackend: GEMM backend the inference layer selects for a given matmul shape.

Constants§

COOP_GEMV_ROWS: Rows per workgroup for multi-row coop GEMV (coop_gemv_mr in fused_transformer.wgsl). Must stay in lock-step with WGSL COOP_ROWS.

Functions§

add_decode_attn_ns: Accumulate one layer’s attention (QKV-proj + SDPA + O-proj) wall-clock.
add_decode_ffn_ns: Accumulate one layer’s FFN (pre-norm SwiGLU) wall-clock.
add_decode_forward_ns: Accumulate one token’s transformer-forward (32-layer) wall-clock.
add_decode_output_ns: Accumulate one token’s output-projection (argmax/top-k) wall-clock.
attention_o_fuse_enabled
attention_preproject_enabled
awq_sweep_blocking: AWQ α-sweep on the ternary FFN (AWQ steps 1–3 end to end): capture activation salience from the Q8 reference at gguf_path, then for each α compile an AWQ-scaled ternary .q42 (compile_gguf_to_q42_ternary_ffn_awq), evaluate its perplexity + unique-word coherence, and return (reference_ppl, [(alpha, ppl, uniq)]). α=0.0 is plain ternary (the baseline). max_tok caps tokens/passage to bound the sweep. Honest: this measures whether AWQ rescues ternary — it does not assume it does. Needs a GPU.
capture_kv_gpu_readback: GPU-readback KV capture for the W5b sparse-dictionary go/no-go — the independent route to the CPU-reference hook. Loads the model with an f32 KV cache, runs the REAL fast GPU decode forward over the eval corpus, and after each passage reads the KV arena back from VRAM ([QTensorEngine::capture_kv_f32]), accumulating up to max_per_layer K and V vectors per layer. Because it samples the actual decode-path vectors (not the CPU reference), it cross-checks the hook capture: if both agree, the measured KV geometry is trustworthy. Stops early once every layer’s cap is hit. Needs a GPU.
compare_topk_decode: A1a correctness: decode the same prompt with the GPU top-k path off then on (same resident model, deterministic argmax) and return both strings. Since k=1 top-k == argmax, the texts must be byte-identical — this verifies the GEMM→top-k wiring, not just the kernel (which is oracle-tested).
compare_topk_decode_blocking: compare_topk_decode inside a fresh multi-thread runtime (residency mount needs block_in_place).
coop_gemv_enabled: Whether native GEMM should run the cooperative coop_gemv kernel rather than the naive 1-thread/row main. Env forces either direction; otherwise the atomic flag (default OFF).
coop_gemv_workgroups: Workgroup count for coop GEMV dispatch: ceil(n_out / COOP_GEMV_ROWS).
coopmat_gemm_enabled: Whether the coopmat GEMM seam is armed (env forces either direction; else the flag). Being armed does not mean coopmat runs — see coopmat_gemm_usable, which also requires the hardware probe.
coopmat_gemm_usable: Whether coopmat is BOTH armed and genuinely usable on this device right now (the forge runtime probe passes — i.e. the wgpu #9741 fix is present). false on wgpu 29.0.3. Feature-guarded: with wgsl-forge off there is no probe, so coopmat is never usable.
cpu_attention_enabled: Whether native attention should use the CPU reference.
cuda_mega_path_counts: (successful CUDA mega-pass forwards, explicit CUDA fallbacks).
decode_attn_ffn: Read the intra-layer attention/FFN accumulators as (attn_ns, ffn_ns) (summed over the run).
decode_budget_fixed_tokens: When budget override is active, ignore EOS so A/B runs a fixed token count (prevents early-stop from inflating/deflating tok/s on short prompts).
decode_budget_override: Current decode-budget override (0 = none).
decode_sampled_blocking: W2: decode with the exact CPU sampler installed for the duration of the call. Returns (text, tok/s). Restores greedy (None) afterwards so it never leaks into other tests.
decode_with_metrics: A1b: mount a model (auto-detecting P64 vs GGUF by magic) and run ONE decode of prompt for decode_tokens, returning (text, decode_tok_s). For a ternary .q42 the FFN routing follows the global set_ternary_ffn toggle, so a caller can measure GPU-ON vs CPU-OFF on identical weights. Caller sets the toggle before invoking. (Use the _blocking wrapper from sync code.)
decode_with_metrics_blocking: decode_with_metrics inside a fresh multi-thread runtime (residency mount needs block_in_place).
empty_rt: Read the empty-round-trip baseline as (total_ns, n).
ffn_f16_enabled: Whether FFN weights should be promoted to f16 in VRAM for decode/prefill GEMV.
ffn_fusion_enabled: Whether the native FFN should run fused (one submit/layer) rather than three GEMM round-trips.
ffn_fusion_in_resident: True when the last-built resident plan wires fused FFN expansion (not just the flag).
gemm_parity_probe_blocking: W3 — GPU↔CPU GEMM parity probe (test/diagnostic). Builds a fresh engine, synthesizes a random Q8_0 weight matrix (n_out rows × n_in; n_in must be a multiple of 32) + input from seed, runs the GPU kernel and the CPU reference on identical bytes, and returns (max_abs_err, mean_abs_err, max_ulp, gpu_gemm_passes_profiled). A non-zero pass count proves the GPU path actually executed — the engine readback falls back to CPU when no tokio handle is present, so the rt.enter() below installs one to force the real GPU path.
gemm_parity_probe_f16_blocking: W3/F16 — GPU↔CPU parity for the new F16 GEMM path (unpack2x16float in the shader vs the CPU dequant_f16 reference). Synthesizes a random F16 weight matrix (n_out rows × n_in; no block constraint) + input from seed, runs both on identical bytes, returns (max_abs_err, mean_abs_err, max_ulp, gpu_gemm_passes). Same witness rule as the Q8 probe.
gpu_topk_enabled: Whether the GPU top-k decode path is active. The env var overrides the flag in BOTH directions (0/false → off, 1/true → on); otherwise the process default (ON) applies.
gpu_wait_count: Re-export: GPU submit → poll(Wait) round-trips counted during the last run (see gguf_bridge).
inference_timeout_ms: Effective timeout: override if non-zero, else env QUALIA_INFERENCE_TIMEOUT_MS, else 30s.
kv_dict_enabled: Whether the KV cache should use the installed sparse dictionary.
kv_int8_enabled: Whether the KV cache should be int8-quantized.
output_path_counts: (top-k hits, argmax fallbacks) since the last reset.
perplexity_eval_blocking: W1 — teacher-forced perplexity of model_path over the eval corpus, run through Qualia’s native engine (never an external runtime). For each corpus passage: reset_kv_cache, then per position embed → dispatch_transformer_forward → apply_output_norm_inplace → dispatch_output_logits_into → NLL of the true next token; PPL = exp(ΣNLL / Σtokens). max_tok = 0 scores the whole passage,
phase_snapshot: Read the phase counters recorded by the last inference call.
record_argmax_fallback: Decode loop: fell back to the full-logit-readback argmax path.
record_cuda_mega_fallback: Decode loop: CUDA mega-pass was explicitly requested but failed eligibility or execution and the ordinary fallback path ran.
record_cuda_mega_hit: Decode loop: the CUDA all-layer mega-pass completed this token forward.
record_decode: Record the autoregressive decode loop time + tokens generated.
record_empty_rt: Record the empty-round-trip baseline: total_ns measured over n empty submit→wait cycles.
record_load_ns: Record model load/mmap-adopt + pipeline-build time (engine ready → before prefill).
record_prefill: Record prefill (prompt KV population) time + tokens prefilled.
record_resident_fallback: Decode loop: resident path was enabled but ineligible/failed — legacy ran instead.
record_resident_hit: Decode loop: the resident single-fence path produced this token.
record_resident_prefill_fallback: Prefill: resident path was enabled but ineligible/failed — legacy chunk ran instead.
record_resident_prefill_hit: Prefill: the resident single-fence-per-chunk arena populated this chunk’s KV.
record_sampled_token: Decode loop: the exact CPU sampler produced this token (non-greedy path).
record_spec_step: One speculative step ran (a draft was proposed + verified).
record_topk_hit: Decode loop: the GPU top-k path produced the next token.
reset_cuda_mega_path_counts: Reset the native CUDA execution counters.
reset_output_path_counts: Reset the output-projection path counters.
reset_phase_metrics: Clear the phase counters before a measured run.
reset_resident_path_counts: Reset the resident-path counters.
reset_resident_prefill_counts: Reset the resident-prefill path counters.
reset_sampled_token_count: Reset the sampled-token counter.
reset_spec_decode_counts: Reset the speculative-decode counters.
resident_decode_enabled: Whether native decode should run the GPU-resident single-fence token path.
resident_path_counts: (resident hits, resident fallbacks) since the last reset.
resident_prefill_counts: (resident-prefill hits, resident-prefill fallbacks) since the last reset.
resident_prefill_enabled: Whether native prefill should run the GPU-resident single-fence-per-chunk arena.
resident_weights_enabled: Whether native GEMM should bind resident per-tensor weight buffers (upload-once) rather than re-uploading the weight every token. Env forces either direction; otherwise the atomic flag.
results_to_csv: CSV (header + one row per result).
results_to_json: Pretty-printed JSON for a result set.
results_to_table: Human-readable table for stdout.
run_bench: Run one benchmark case end-to-end (cold then warm) against the real path.
run_suite: Run a suite of cases, skipping any whose model file is absent.
run_suite_blocking: Run a suite inside a fresh multi-thread Tokio runtime.
sampled_token_count: Sampled tokens since the last reset.
sampler_config: The active decode sampler config, if a non-greedy one is installed.
select_gemm_backend: Select the GEMM backend for an m×k×n matmul: coopmat when armed+usable and all dims are 8-mult (the tensor-core tile — batched prefill, not the m=1 decode GEMV); else cooperative GEMV when enabled; else naive. Pure + total, so it is unit-tested without a GPU.
set_attention_o_fuse
set_attention_preproject
set_coop_gemv: Enable/disable the cooperative GEMV decode path (QUALIA_LLM_COOP_GEMV).
set_coopmat_gemm: Arm/disarm the coopmat (tensor-core) GEMM selection seam (QUALIA_LLM_COOPMAT).
set_cpu_attention: Enable/disable the native CPU-attention reference path (QUALIA_LLM_CPU_ATTENTION).
set_decode_budget_override: Set a fixed decode-token budget for benchmarking (0 = production default).
set_ffn_f16: Enable/disable FFN quant→f16 promotion at resident-plan build (QUALIA_LLM_FFN_F16).
set_ffn_fusion: Enable/disable the fused single-submit FFN path (QUALIA_LLM_FFN_FUSION).
set_ffn_fusion_in_resident
set_gpu_topk: Enable/disable the GPU top-k decode path (QUALIA_LLM_GPU_TOPK).
set_inference_timeout_override_ms: Set wall-clock decode timeout in ms (0 = production default 30_000).
set_kv_dict: Enable/disable the sparse-dictionary KV cache (QUALIA_LLM_KV_DICT).
set_kv_int8: Enable/disable the int8 KV cache (QUALIA_LLM_KV_INT8).
set_resident_decode: Enable/disable the resident-token single-fence decode (QUALIA_LLM_RESIDENT_DECODE).
set_resident_prefill: Enable/disable the resident single-fence-per-chunk prefill path (QUALIA_LLM_RESIDENT_PREFILL).
set_resident_weights: Enable/disable the resident per-tensor weight buffers (QUALIA_LLM_RESIDENT_WEIGHTS).
set_sampler_config: Install the decode sampler config (None restores greedy argmax).
set_spec_decode: Enable/disable prompt-lookup speculative decode (QUALIA_LLM_SPEC_DECODE). Runtime mode switch — the desktop UI / host calls this to flip between exact single-token decode and speculative decode.
set_ternary_ffn: Enable/disable the resident 2-bit GPU ternary-FFN path (QUALIA_LLM_TERNARY_FFN).
spec_decode_counts: (spec steps, tokens drafted, draft tokens accepted) since the last reset.
spec_decode_enabled: Whether the decode loop should run prompt-lookup speculative decode (the effective mode: env var wins if set, else the runtime flag). Read this to reflect the current mode in a UI.
spec_verify_probe_blocking: W6a — batched verify-primitive correctness probe. Prefills prompt (positions [0, p), p = prompt_len-1), snapshots the KV cache, then:
ternary_ffn_enabled: Whether the GPU ternary-FFN path is active (atomic flag OR the env var). When false, ternary FFN GEMMs fall back to the CPU oracle (correct, slower) — the toggle’s OFF baseline.