pub struct SchedulingPolicy {
pub max_prefill_chunk_tokens: u32,
pub max_consecutive_prefill_chunks: u32,
}Expand description
Policy governing chunk limits and prefill-vs-decode interleaving.
Fields§
§max_prefill_chunk_tokens: u32Maximum tokens processed in a single prefill chunk.
max_consecutive_prefill_chunks: u32Maximum consecutive prefill chunks permitted while decode requests are waiting.
Implementations§
Source§impl SchedulingPolicy
impl SchedulingPolicy
pub const DEFAULT: Self
Sourcepub fn new(
max_prefill_chunk_tokens: u32,
max_consecutive_prefill_chunks: u32,
) -> Self
pub fn new( max_prefill_chunk_tokens: u32, max_consecutive_prefill_chunks: u32, ) -> Self
Creates a policy with prefill-chunk size and consecutive-chunk cap.
Sourcepub fn select_chunk_size(&self, remaining_tokens: u32) -> u32
pub fn select_chunk_size(&self, remaining_tokens: u32) -> u32
Select chunk length given remaining prompt tokens.
Sourcepub fn should_yield_prefill_to_decode(
&self,
consecutive_prefill_count: u32,
has_waiting_decodes: bool,
) -> bool
pub fn should_yield_prefill_to_decode( &self, consecutive_prefill_count: u32, has_waiting_decodes: bool, ) -> bool
Decide whether prefill must yield to waiting decode requests.
Trait Implementations§
Source§impl Clone for SchedulingPolicy
impl Clone for SchedulingPolicy
impl Copy for SchedulingPolicy
Source§impl Debug for SchedulingPolicy
impl Debug for SchedulingPolicy
Source§impl Default for SchedulingPolicy
impl Default for SchedulingPolicy
impl Eq for SchedulingPolicy
Source§impl PartialEq for SchedulingPolicy
impl PartialEq for SchedulingPolicy
impl StructuralPartialEq for SchedulingPolicy
Auto Trait Implementations§
impl Freeze for SchedulingPolicy
impl RefUnwindSafe for SchedulingPolicy
impl Send for SchedulingPolicy
impl Sync for SchedulingPolicy
impl Unpin for SchedulingPolicy
impl UnsafeUnpin for SchedulingPolicy
impl UnwindSafe for SchedulingPolicy
Blanket Implementations§
§impl<S, A> Aggregate<Result<S, Error>> for Awhere
A: Aggregate<S>,
impl<S, A> Aggregate<Result<S, Error>> for Awhere
A: Aggregate<S>,
Aggregate shares in an MPC protocol.
Source§impl<T> BorrowMut<T> for Twhere
T: ?Sized,
impl<T> BorrowMut<T> for Twhere
T: ?Sized,
Source§fn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
Mutably borrows from an owned value. Read more
impl<ST, DT> CastableFrom<ST, Initialized, Initialized> for DT
impl<ST, DT> CastableFrom<ST, Uninit, Uninit> for DT
Source§impl<T> CloneToUninit for Twhere
T: Clone,
impl<T> CloneToUninit for Twhere
T: Clone,
§impl<Q, K> Equivalent<K> for Q
impl<Q, K> Equivalent<K> for Q
§fn equivalent(&self, key: &K) -> bool
fn equivalent(&self, key: &K) -> bool
Checks if this value is equivalent to the given key. Read more
§impl<Q, K> Equivalent<K> for Q
impl<Q, K> Equivalent<K> for Q
§fn equivalent(&self, key: &K) -> bool
fn equivalent(&self, key: &K) -> bool
Compare self to
key and return true if they are equal.§impl<Q, K> Equivalent<K> for Q
impl<Q, K> Equivalent<K> for Q
§fn equivalent(&self, key: &K) -> bool
fn equivalent(&self, key: &K) -> bool
Checks if this value is equivalent to the given key. Read more
§impl<T> Instrument for T
impl<T> Instrument for T
§fn instrument(self, span: Span) -> Instrumented<Self> ⓘ
fn instrument(self, span: Span) -> Instrumented<Self> ⓘ
Source§impl<T> IntoEither for T
impl<T> IntoEither for T
Source§fn into_either(self, into_left: bool) -> Either<Self, Self> ⓘ
fn into_either(self, into_left: bool) -> Either<Self, Self> ⓘ
Converts
self into a Left variant of Either<Self, Self>
if into_left is true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read moreSource§fn into_either_with<F>(self, into_left: F) -> Either<Self, Self> ⓘ
fn into_either_with<F>(self, into_left: F) -> Either<Self, Self> ⓘ
Converts
self into a Left variant of Either<Self, Self>
if into_left(&self) returns true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read more§impl<T> NoneValue for Twhere
T: Default,
impl<T> NoneValue for Twhere
T: Default,
type NoneType = T
§fn null_value() -> T
fn null_value() -> T
The none-equivalent value.