Expand description
Bounded request scheduling for paged native inference.
Construction is cold and sizes every request table. Admission, prefix attachment, runnable collection, page COW, cancellation, and completion are zero-growth hot operations.
Re-exports§
pub use admission::AdmissionError;pub use admission::AdmittedCurrencies;pub use drain::DrainController;pub use drain::DrainError;pub use drain::DrainReason;pub use drain::DrainState;pub use policy::PrefillProgress;pub use policy::SchedulingPolicy;pub use queue::BoundedIntakeQueue;pub use queue::IntakeError;pub use queue::IntakeRequest;pub use queue::IntakeState;pub use ragged_backend::select_graph_bucket;pub use ragged_backend::MultiSequenceRaggedBackend;pub use ragged_backend::RaggedBackendStats;pub use ragged_backend::GRAPH_BUCKET_SIZES;pub use ragged_backend::MAX_BATCH_BUCKET;pub use ragged_backend::PADDED_REQUEST_ID;pub use ragged_backend::PADDED_SLOT_ID;
Modules§
- admission
- Currency reservation and strict pre-admission gating.
- drain
- Safe maintenance, idle-drain state machine, and generation invalidation (Work Package F7).
- policy
- Deterministic prefill and decode scheduling policies.
- queue
- Bounded request queue and worker intake lifecycle.
- ragged_
backend - Multi-sequence ragged decode backend with CUDA graph replay bucketing (Work Package F6).
Structs§
- Admission
- Ragged
Batch Item - Ragged
Batch Output - Ragged
Batch Receipt - Request
Scheduler - Fixed-request-capacity scheduler.
REQUESTSis the hard concurrent request bound. - Request
View
Enums§
Traits§
- Ragged
Decode Backend - Prepared backend boundary for one continuous-batching decode round.