qualia_core_db/inference/inference_bench/
mod.rs

1//! A0 — native LLM benchmark harness (STELLAR §A; decisions D17 + D22).
2//!
3//! This is the **shared** measurement surface for the performance push. It drives
4//! the *real* inference path ([`LocalLlmAgent::infer_local_model_streaming`]) and
5//! reads per-phase timing recorded *inside that same path* — so the existing
6//! F16/Q8 path and the future ternary/top-k paths are measured by **one** harness
7//! rather than a forked benchmark loop (D22 "shared-improvement" rule). A speedup
8//! that shows up here is a real, attributable, end-to-end number, not a kernel
9//! microbenchmark.
10//!
11//! What it reports (per model / weight policy):
12//!   * **cold TTFT** — model not resident: wall-clock from call to first token
13//!     (bundles mmap load + pipeline create + prefill + first decode);
14//!   * **warm TTFT** — model resident (mmap adopted, pipelines still rebuilt per
15//!     call in the current architecture — that cost is intentionally *included*,
16//!     it is what A7 will attack);
17//!   * **prefill / decode tok/s** from the internal phase split;
18//!   * the **load / prefill / decode** wall-clock breakdown.
19//!
20//! Honest scope of *this* increment (A0.1): timings are **host wall-clock**.
21//! GPU timestamp-query kernel isolation (D17) — requesting `TIMESTAMP_QUERY` on
22//! the shared device and wrapping passes with `timestamp_writes` — is the A0.2
23//! follow-on; [`BenchResult::gpu_timestamp_supported`] is `false` until then.
24//!
25//! Native-only: the WASM decode path is a different beast and is benchmarked in
26//! the browser harness.
27//!
28//! Library-ized (CLAUDE.md §11) from a single `inference_bench.rs` into cohesive
29//! submodules — pure code motion, no behaviour change. The public surface is
30//! re-exported unchanged, so `crate::inference::inference_bench::<Item>` resolves
31//! exactly as before.
32#![cfg(not(target_arch = "wasm32"))]
33
34mod counters;
35mod metrics;
36mod probes;
37pub mod raw_decode;
38mod reporting;
39mod runner;
40mod toggles;
41mod types;
42
43#[cfg(test)]
44mod tests;
45
46pub use counters::*;
47pub use metrics::*;
48pub use probes::*;
49pub use raw_decode::*;
50pub use reporting::*;
51pub use runner::*;
52pub use toggles::*;
53pub use types::*;
qualia_core_db/inference/inference_bench/mod.rs

qualia_core_db/inference/inference_bench/
mod.rs
