Skip to main content

Module runtime

Module runtime 

Source
Expand description

Prepared native inference runtime.

This library is the plan/run boundary for production inference. Model discovery, compilation, immutable upload, receipts, and artifact management are cold control-plane work. A prepared backend’s decode step is the Tier-1 zero-heap data plane.

Re-exports§

pub use memory_guard::HostMemorySample;
pub use memory_guard::MemoryPolicyError;
pub use memory_guard::MemoryPressureAction;
pub use memory_guard::MemoryPressureGuard;
pub use memory_guard::MemoryPressurePolicy;
pub use artifacts::cleanup_stale_runs;
pub use artifacts::ArtifactError;
pub use artifacts::ArtifactFinish;
pub use artifacts::ArtifactRetention;
pub use artifacts::ArtifactStats;
pub use artifacts::RunArtifactDir;
pub use artifacts::StaleCleanupReport;
pub use prepared::DecodeStepError;
pub use prepared::DecodeStepInput;
pub use prepared::DecodeStepOutput;
pub use prepared::PreparedBackend;
pub use prepared::PreparedDecodePlan;
pub use prepared::PreparedPlanState;
pub use receipt::capture_source_provenance;
pub use receipt::sha256_file;
pub use receipt::sha256_token_ids;
pub use receipt::BenchmarkManifest;
pub use receipt::SourceProvenance;
pub use receipt::MANIFEST_SCHEMA_VERSION;
pub use receipt::RAW_GREEDY_DECODE_POLICY;
pub use scheduler::Admission;
pub use scheduler::RequestScheduler;
pub use scheduler::RequestState;
pub use scheduler::RequestView;
pub use scheduler::SchedulerError;

Modules§

artifacts
Bounded temporary artifacts and explicit evidence promotion.
budget
Joint memory budget planning and checked arithmetic.
graph_assist
Explicit graph-assisted inference ABI.
kv
KV-cache planning and ownership.
memory_guard
Fail-closed host-memory pressure policy for native inference.
prepared
Backend-neutral prepared decode contract.
receipt
Serializable cold-path evidence and fixed-size hot-path counters.
scheduler
Bounded request scheduling for paged native inference.
stage

Structs§

ArtifactCleanupCounters
BlockGeometry
Geometry of a physical KV cache block/page.
ExecutionCounters
ExecutionReceipt
MemoryPoolBudget
Joint memory plan configuration across device VRAM and pools.
ModelMemoryProfile
Static memory profile of a loaded model on a device backend.
RequestReservation
An admitted request’s checked memory reservation.
StageTimingReceipt
Fixed-size per-stage timing and outcome record. auxiliary_code carries stage-specific counts (e.g. routed expert count, SSM state steps, or deliberation tokens produced).
StageTrace
Caller-buffered, zero-heap recorder for one inference turn’s stage timing.
ThinkingTracker
Zero-heap streaming detector for thinking/deliberation regions in decode output, with a token count for budget enforcement.

Enums§

BackendKind
BudgetError
Errors returned during budget computation and admission reservation.
InferenceStageKind
Lifecycle stage kind. Discriminants match the ordering in inference-lifecycle.n3 §3 (Qualia stages then model-native stages).
StageStatus
Terminal disposition of a stage within one inference turn.
ThinkingEvent
Event returned by ThinkingTracker::feed_token.

Constants§

RECEIPT_SCHEMA_VERSION
STAGE_TRACE_CAPACITY
Number of stage receipts an StageTrace (and an super::receipt::ExecutionReceipt) can hold. There are 15 stage kinds, so a per-kind-coalesced trace never overflows.

Functions§

last_stage_trace
Read the most recently published stage trace (empty before any decode).
set_last_stage_trace
Publish a completed per-turn stage trace.
take_last_stage_trace
Read and clear the most recently published stage trace.