Expand description
Prepared native inference runtime.
This library is the plan/run boundary for production inference. Model discovery, compilation, immutable upload, receipts, and artifact management are cold control-plane work. A prepared backend’s decode step is the Tier-1 zero-heap data plane.
Re-exports§
pub use memory_guard::HostMemorySample;pub use memory_guard::MemoryPolicyError;pub use memory_guard::MemoryPressureAction;pub use memory_guard::MemoryPressureGuard;pub use memory_guard::MemoryPressurePolicy;pub use artifacts::cleanup_stale_runs;pub use artifacts::ArtifactError;pub use artifacts::ArtifactFinish;pub use artifacts::ArtifactRetention;pub use artifacts::ArtifactStats;pub use artifacts::RunArtifactDir;pub use artifacts::StaleCleanupReport;pub use prepared::DecodeStepError;pub use prepared::DecodeStepInput;pub use prepared::DecodeStepOutput;pub use prepared::PreparedBackend;pub use prepared::PreparedDecodePlan;pub use prepared::PreparedPlanState;pub use receipt::capture_source_provenance;pub use receipt::sha256_file;pub use receipt::sha256_token_ids;pub use receipt::BenchmarkManifest;pub use receipt::SourceProvenance;pub use receipt::MANIFEST_SCHEMA_VERSION;pub use receipt::RAW_GREEDY_DECODE_POLICY;pub use scheduler::Admission;pub use scheduler::RequestScheduler;pub use scheduler::RequestState;pub use scheduler::RequestView;pub use scheduler::SchedulerError;
Modules§
- artifacts
- Bounded temporary artifacts and explicit evidence promotion.
- budget
- Joint memory budget planning and checked arithmetic.
- graph_
assist - Explicit graph-assisted inference ABI.
- kv
- KV-cache planning and ownership.
- memory_
guard - Fail-closed host-memory pressure policy for native inference.
- prepared
- Backend-neutral prepared decode contract.
- receipt
- Serializable cold-path evidence and fixed-size hot-path counters.
- scheduler
- Bounded request scheduling for paged native inference.
- stage
Structs§
- Artifact
Cleanup Counters - Block
Geometry - Geometry of a physical KV cache block/page.
- Execution
Counters - Execution
Receipt - Memory
Pool Budget - Joint memory plan configuration across device VRAM and pools.
- Model
Memory Profile - Static memory profile of a loaded model on a device backend.
- Request
Reservation - An admitted request’s checked memory reservation.
- Stage
Timing Receipt - Fixed-size per-stage timing and outcome record.
auxiliary_codecarries stage-specific counts (e.g. routed expert count, SSM state steps, or deliberation tokens produced). - Stage
Trace - Caller-buffered, zero-heap recorder for one inference turn’s stage timing.
- Thinking
Tracker - Zero-heap streaming detector for thinking/deliberation regions in decode output, with a token count for budget enforcement.
Enums§
- Backend
Kind - Budget
Error - Errors returned during budget computation and admission reservation.
- Inference
Stage Kind - Lifecycle stage kind. Discriminants match the ordering in
inference-lifecycle.n3§3 (Qualia stages then model-native stages). - Stage
Status - Terminal disposition of a stage within one inference turn.
- Thinking
Event - Event returned by
ThinkingTracker::feed_token.
Constants§
- RECEIPT_
SCHEMA_ VERSION - STAGE_
TRACE_ CAPACITY - Number of stage receipts an
StageTrace(and ansuper::receipt::ExecutionReceipt) can hold. There are 15 stage kinds, so a per-kind-coalesced trace never overflows.
Functions§
- last_
stage_ trace - Read the most recently published stage trace (empty before any decode).
- set_
last_ stage_ trace - Publish a completed per-turn stage trace.
- take_
last_ stage_ trace - Read and clear the most recently published stage trace.