Skip to main content

Module scheduler

Module scheduler 

Source
Expand description

Bounded request scheduling for paged native inference.

Construction is cold and sizes every request table. Admission, prefix attachment, runnable collection, page COW, cancellation, and completion are zero-growth hot operations.

Re-exports§

pub use admission::AdmissionError;
pub use admission::AdmittedCurrencies;
pub use drain::DrainController;
pub use drain::DrainError;
pub use drain::DrainReason;
pub use drain::DrainState;
pub use policy::PrefillProgress;
pub use policy::SchedulingPolicy;
pub use queue::BoundedIntakeQueue;
pub use queue::IntakeError;
pub use queue::IntakeRequest;
pub use queue::IntakeState;
pub use ragged_backend::select_graph_bucket;
pub use ragged_backend::MultiSequenceRaggedBackend;
pub use ragged_backend::RaggedBackendStats;
pub use ragged_backend::GRAPH_BUCKET_SIZES;
pub use ragged_backend::MAX_BATCH_BUCKET;
pub use ragged_backend::PADDED_REQUEST_ID;
pub use ragged_backend::PADDED_SLOT_ID;

Modules§

admission
Currency reservation and strict pre-admission gating.
drain
Safe maintenance, idle-drain state machine, and generation invalidation (Work Package F7).
policy
Deterministic prefill and decode scheduling policies.
queue
Bounded request queue and worker intake lifecycle.
ragged_backend
Multi-sequence ragged decode backend with CUDA graph replay bucketing (Work Package F6).

Structs§

Admission
RaggedBatchItem
RaggedBatchOutput
RaggedBatchReceipt
RequestScheduler
Fixed-request-capacity scheduler. REQUESTS is the hard concurrent request bound.
RequestView

Enums§

DecodeRoundError
RaggedBackendError
RequestState
SchedulerError

Traits§

RaggedDecodeBackend
Prepared backend boundary for one continuous-batching decode round.