Expand description
Inference superiority lab — instruments for evidence-based methodology.
Plan: docs/plans/inference-superiority-lab-and-toolset-plan.md
Native-only. No Ollama product dependency; optional external A/B via CLI script.
Re-exports§
pub use ablate::run_ablation_matrix;pub use audit_path::audit_hot_path;pub use audit_path::HotPathAudit;pub use auto_improve::default_search_space;pub use auto_improve::run_auto_improve;pub use auto_improve::LabConfig;pub use auto_improve::LockInPackage;pub use auto_improve::TrialResult;pub use device_roof::calibrate_device_roof;pub use micro::run_q4k_soa_microbench;pub use operator_promotion::CapabilityMatrix;pub use operator_promotion::OperatorPromotionReceipt;pub use operator_promotion::PromotionError;pub use operator_promotion::PromotionStatus;pub use operator_promotion::SupportLevel;pub use timeline::run_decode_timeline;pub use campaign::run_optimization_campaign;pub use experiment::run_experiment;pub use experiment::run_experiment_with_quality;
Modules§
- ablate
- audit_
path - Hot-path wiring auditor — makes unfinished integration visible.
- auto_
improve - campaign
- config_
space - device_
roof - Device peak calibration — real roofline anchors (plan L1.2).
- experiment
- Experiment execution engine — wraps the existing
run_benchharness with quality verification, thermal sampling, and structured result collection. - experiment_
log - hypothesis
- micro
- Isolated Q4_K SoA GEMV microbench (CPU oracle + optional CUDA).
- operator_
compare - Canonical comparison execution with host-specific checkpoint discovery tests.
- operator_
metrics - Compatibility forwarding to canonical operator experiment receipts.
- operator_
promotion - Operator promotion receipts and capability support matrix (W9: EOS-090).
- pareto
- search
- timeline
- Decode timeline: host phase counters + optional GPU profile + tok/s.
Structs§
- Ablation
Row - Auto
Improve Config - Batch
Comparison Result - Batch scaling measurement row for EOS-042.
- Belief
Graph - The typed belief graph.
- Bench
Result Serde - Serializable subset of
BenchResult(the original only derivesSerialize). - Campaign
Config - Configuration for an optimization campaign.
- Campaign
Report - The result of an optimization campaign.
- Configuration
- A concrete configuration: one point in the space.
- Configuration
Space - A configuration space: named parameters with definitions.
- Decode
Timeline - Device
Roof - Experiment
Config - Configuration for one experiment run.
- Experiment
Result - The full result of one experiment run.
- Experiment
Run - FfnBlock
Weights - SwiGLU FFN block weights in packed GGML Q4_K format.
- Hypothesis
- A hypothesis: a testable claim about an optimization strategy.
- KnnSurrogate
- A simple k-nearest-neighbor surrogate model for the objective function. Predicts the objective at a new point as the distance-weighted average of the k nearest observed points. This is a lightweight alternative to a full Random Forest (SMAC3) or Gaussian Process, suitable for small budgets.
- Microbench
Result - Operator
Experiment Receipt - Structured lab experiment receipt recording operator conversion, execution, and fidelity.
- Operator
Memory Breakdown - Memory footprint breakdown across separate memory domains.
- Pareto
Frontier - A Pareto frontier: the set of non-dominated points from a collection of experiments.
- Pareto
Point - The six Pareto dimensions. “Lower is better” dimensions are negated internally so that “higher is always better” in the dominance comparison.
- Phase
Snapshot Serde - Serializable phase snapshot (the atomic counters are copied into plain u64s).
- Quality
Score - Quality score derived from
VerifiedTurnchecks. - Search
Engine - The search engine: combines Sobol initial design with k-NN + EI Bayesian optimization.
- Sobol
Sequence - A Sobol quasi-random sequence generator for uniform coverage of the [0,1]^d unit hypercube. Uses the Joe & Kuo direction numbers (compact implementation).
- Thermal
Snapshot - Thermal reading before and after an experiment.
- Track
AndStop Bandit - A multi-armed bandit using the Track-and-Stop algorithm for best-arm identification. Given a small set of pre-selected configurations, it adaptively allocates samples to identify the best one with statistical rigor.
Enums§
- Application
Profile Weight - Score a result against an application profile to select the “best” point from the Pareto frontier.
- Experiment
Verdict - The verdict of an experiment on a hypothesis.
- Parameter
Def - One parameter definition in the configuration space.
- Parameter
Value - A concrete parameter value.
Constants§
Functions§
- append_
experiment_ jsonl - append_
run_ csv - Append one run; create file with header if missing.
- compare_
batch_ scaling - Run multi-batch scaling comparison across batch sizes [1, 2, 4, 8] (EOS-042).
- compare_
single_ tensor - Run single tensor comparison and return an experiment receipt (EOS-041).
- default_
toggle_ space - Build a default configuration space for inference toggle optimization.
This covers the main toggle parameters that
ExperimentConfig::apply_togglessupports. - evaluate_
verdict - Determine the verdict of an experiment against a hypothesis. This compares the experiment result against a baseline.
- execute_
ffn_ existing - Execute a full SwiGLU FFN block: down(silu(gate(x)) * up(x)) using existing Q4_K dequant.
- execute_
ffn_ lookup - Execute a full SwiGLU FFN block using the new executable operator lookup runtime.
- expected_
improvement - Expected Improvement (EI) acquisition function. EI(x) = (mu - f_best) * Phi(Z) + sigma * phi(Z) where Z = (mu - f_best) / sigma. Higher EI = more promising to try.
- format_
lockin_ summary - Renders a lock-in package summary.
- load_
experiment_ log - save_
campaign_ report - Save a campaign report to a JSON file.