pub struct RequestScheduler<const REQUESTS: usize> { /* private fields */ }Expand description
Fixed-request-capacity scheduler. REQUESTS is the hard concurrent request bound.
Implementations§
Source§impl<const REQUESTS: usize> RequestScheduler<REQUESTS>
impl<const REQUESTS: usize> RequestScheduler<REQUESTS>
Sourcepub fn new(logical_pages: u32) -> Self
pub fn new(logical_pages: u32) -> Self
Cold construction; allocates each fixed-capacity logical page table exactly once.
Sourcepub fn with_generation(logical_pages: u32, initial_generation: u64) -> Self
pub fn with_generation(logical_pages: u32, initial_generation: u64) -> Self
Creates a scheduler over logical_pages starting at initial_generation.
Sourcepub fn drain(&self) -> &DrainController
pub fn drain(&self) -> &DrainController
Drain controller for graceful shutdown.
Sourcepub fn drain_mut(&mut self) -> &mut DrainController
pub fn drain_mut(&mut self) -> &mut DrainController
Mutable drain controller.
Sourcepub fn request_drain(
&mut self,
reason: DrainReason,
current_epoch: u64,
) -> Result<DrainState, DrainError>
pub fn request_drain( &mut self, reason: DrainReason, current_epoch: u64, ) -> Result<DrainState, DrainError>
Initiates/updates drain state.
Sourcepub fn apply_memory_pressure(
&mut self,
guard: &MemoryPressureGuard,
sample: HostMemorySample,
current_epoch: u64,
) -> MemoryPressureAction
pub fn apply_memory_pressure( &mut self, guard: &MemoryPressureGuard, sample: HostMemorySample, current_epoch: u64, ) -> MemoryPressureAction
Apply a host-memory policy sample before a new scheduling round.
At critical pressure this transitions the scheduler into its existing drain state, so active requests finish only through normal token-boundary lifecycle handling. The caller performs cache shedding/checkpointing for the returned action; this hot path remains allocation-free.
Sourcepub fn active_count(&self) -> usize
pub fn active_count(&self) -> usize
Currently active requests.
Sourcepub fn admit_with_prefix<const ENTRIES: usize, const PAGES: usize>(
&mut self,
request_id: u64,
identity: Option<PrefixIdentity>,
prefixes: &PrefixKvStore<ENTRIES, PAGES>,
pool: &mut BlockPool,
) -> Result<Admission, SchedulerError>
pub fn admit_with_prefix<const ENTRIES: usize, const PAGES: usize>( &mut self, request_id: u64, identity: Option<PrefixIdentity>, prefixes: &PrefixKvStore<ENTRIES, PAGES>, pool: &mut BlockPool, ) -> Result<Admission, SchedulerError>
Admit a request and atomically attach graph-derived prefix pages when available.
Sourcepub fn ensure_writable(
&mut self,
request_id: u64,
logical_page: u32,
pool: &mut BlockPool,
) -> Result<CopyOnWrite, SchedulerError>
pub fn ensure_writable( &mut self, request_id: u64, logical_page: u32, pool: &mut BlockPool, ) -> Result<CopyOnWrite, SchedulerError>
Make a logical page writable. A Copy result instructs the backend to copy the physical
KV page before writing the next token.
Sourcepub fn mark_prefill_complete(
&mut self,
request_id: u64,
) -> Result<(), SchedulerError>
pub fn mark_prefill_complete( &mut self, request_id: u64, ) -> Result<(), SchedulerError>
Transitions request_id out of prefill; error when unknown/out-of-state.
Sourcepub fn seed_decode_token(
&mut self,
request_id: u64,
token_id: u32,
) -> Result<(), SchedulerError>
pub fn seed_decode_token( &mut self, request_id: u64, token_id: u32, ) -> Result<(), SchedulerError>
Publish the token consumed by the request’s next decode step.
Sourcepub fn record_token(&mut self, request_id: u64) -> Result<u32, SchedulerError>
pub fn record_token(&mut self, request_id: u64) -> Result<u32, SchedulerError>
Records one decoded token for request_id; returns its sequence position.
Sourcepub fn runnable_into(
&self,
out: &mut [RequestView],
) -> Result<usize, SchedulerError>
pub fn runnable_into( &self, out: &mut [RequestView], ) -> Result<usize, SchedulerError>
Write all runnable requests into caller storage in stable slot order.
Sourcepub fn execute_decode_round<B: RaggedDecodeBackend>(
&mut self,
backend: &mut B,
item_scratch: &mut [RaggedBatchItem],
block_table_scratch: &mut [u32],
output_scratch: &mut [RaggedBatchOutput],
) -> Result<RaggedBatchReceipt, DecodeRoundError>
pub fn execute_decode_round<B: RaggedDecodeBackend>( &mut self, backend: &mut B, item_scratch: &mut [RaggedBatchItem], block_table_scratch: &mut [u32], output_scratch: &mut [RaggedBatchOutput], ) -> Result<RaggedBatchReceipt, DecodeRoundError>
Execute one ragged decode round with exactly one backend call.
Active decode requests are lowered in stable scheduler-slot order. Their logical page
tables are concatenated into block_table_scratch; each item carries its table offset.
Outputs are identity-checked as a complete batch before scheduler state is mutated.
Auto Trait Implementations§
impl<const REQUESTS: usize> Freeze for RequestScheduler<REQUESTS>
impl<const REQUESTS: usize> RefUnwindSafe for RequestScheduler<REQUESTS>
impl<const REQUESTS: usize> Send for RequestScheduler<REQUESTS>
impl<const REQUESTS: usize> Sync for RequestScheduler<REQUESTS>
impl<const REQUESTS: usize> Unpin for RequestScheduler<REQUESTS>
impl<const REQUESTS: usize> UnsafeUnpin for RequestScheduler<REQUESTS>
impl<const REQUESTS: usize> UnwindSafe for RequestScheduler<REQUESTS>
Blanket Implementations§
§impl<S, A> Aggregate<Result<S, Error>> for Awhere
A: Aggregate<S>,
impl<S, A> Aggregate<Result<S, Error>> for Awhere
A: Aggregate<S>,
Source§impl<T> BorrowMut<T> for Twhere
T: ?Sized,
impl<T> BorrowMut<T> for Twhere
T: ?Sized,
Source§fn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
impl<ST, DT> CastableFrom<ST, Initialized, Initialized> for DT
impl<ST, DT> CastableFrom<ST, Uninit, Uninit> for DT
§impl<T> Instrument for T
impl<T> Instrument for T
§fn instrument(self, span: Span) -> Instrumented<Self> ⓘ
fn instrument(self, span: Span) -> Instrumented<Self> ⓘ
Source§impl<T> IntoEither for T
impl<T> IntoEither for T
Source§fn into_either(self, into_left: bool) -> Either<Self, Self> ⓘ
fn into_either(self, into_left: bool) -> Either<Self, Self> ⓘ
self into a Left variant of Either<Self, Self>
if into_left is true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read moreSource§fn into_either_with<F>(self, into_left: F) -> Either<Self, Self> ⓘ
fn into_either_with<F>(self, into_left: F) -> Either<Self, Self> ⓘ
self into a Left variant of Either<Self, Self>
if into_left(&self) returns true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read more