Struct MemoryPoolBudget
pub struct MemoryPoolBudget {
pub total_device_bytes: u64,
pub model_profile: ModelMemoryProfile,
pub geometry: BlockGeometry,
pub max_physical_blocks: u32,
pub max_cow_transient_blocks: u32,
}Expand description
Joint memory plan configuration across device VRAM and pools.
Fields§
§total_device_bytes: u64Total device memory budget (e.g. 12 GB on RTX A2000).
model_profile: ModelMemoryProfileStatic model memory profile.
geometry: BlockGeometryPage block geometry.
max_physical_blocks: u32Physical block pool limit.
max_cow_transient_blocks: u32Maximum concurrent COW transient blocks reserved across active requests.
Implementations§
§impl MemoryPoolBudget
impl MemoryPoolBudget
pub fn new(
total_device_bytes: u64,
model_profile: ModelMemoryProfile,
geometry: BlockGeometry,
max_physical_blocks: u32,
max_cow_transient_blocks: u32,
) -> Result<MemoryPoolBudget, BudgetError>
pub fn new( total_device_bytes: u64, model_profile: ModelMemoryProfile, geometry: BlockGeometry, max_physical_blocks: u32, max_cow_transient_blocks: u32, ) -> Result<MemoryPoolBudget, BudgetError>
Creates the budget.
pub fn available_kv_bytes(&self) -> u64
pub fn available_kv_bytes(&self) -> u64
Compute available dynamic memory for KV pools after static model requirements.
§impl MemoryPoolBudget
impl MemoryPoolBudget
pub fn reserve(
&self,
input_tokens: u32,
reserved_output_tokens: u32,
cow_blocks: u32,
active_reserved_bytes: u64,
) -> Result<RequestReservation, BudgetError>
pub fn reserve( &self, input_tokens: u32, reserved_output_tokens: u32, cow_blocks: u32, active_reserved_bytes: u64, ) -> Result<RequestReservation, BudgetError>
Check whether a request can be admitted within the pool budget.
Evaluates:
input_tokens + reserved_output_tokens(checked arithmetic)- Ceiling block conversion via block geometry (fragmented page boundary)
- COW transient headroom allocation
- Global physical block limit
- Global dynamic KV byte ceiling Check whether a request can be admitted within the pool budget without active COW tracking.
pub fn reserve_with_active_cow(
&self,
input_tokens: u32,
reserved_output_tokens: u32,
cow_blocks: u32,
active_cow_blocks: u32,
active_reserved_bytes: u64,
) -> Result<RequestReservation, BudgetError>
pub fn reserve_with_active_cow( &self, input_tokens: u32, reserved_output_tokens: u32, cow_blocks: u32, active_cow_blocks: u32, active_reserved_bytes: u64, ) -> Result<RequestReservation, BudgetError>
Check whether a request can be admitted within the pool budget with cumulative active COW headroom tracking.
Trait Implementations§
§impl Clone for MemoryPoolBudget
impl Clone for MemoryPoolBudget
§fn clone(&self) -> MemoryPoolBudget
fn clone(&self) -> MemoryPoolBudget
Returns a duplicate of the value. Read more
1.0.0 (const: unstable) · Source§fn clone_from(&mut self, source: &Self)
fn clone_from(&mut self, source: &Self)
Performs copy-assignment from
source. Read more§impl Debug for MemoryPoolBudget
impl Debug for MemoryPoolBudget
impl Eq for MemoryPoolBudget
§impl PartialEq for MemoryPoolBudget
impl PartialEq for MemoryPoolBudget
impl StructuralPartialEq for MemoryPoolBudget
Auto Trait Implementations§
impl Freeze for MemoryPoolBudget
impl RefUnwindSafe for MemoryPoolBudget
impl Send for MemoryPoolBudget
impl Sync for MemoryPoolBudget
impl Unpin for MemoryPoolBudget
impl UnsafeUnpin for MemoryPoolBudget
impl UnwindSafe for MemoryPoolBudget
Blanket Implementations§
§impl<S, A> Aggregate<Result<S, Error>> for Awhere
A: Aggregate<S>,
impl<S, A> Aggregate<Result<S, Error>> for Awhere
A: Aggregate<S>,
Aggregate shares in an MPC protocol.
Source§impl<T> BorrowMut<T> for Twhere
T: ?Sized,
impl<T> BorrowMut<T> for Twhere
T: ?Sized,
Source§fn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
Mutably borrows from an owned value. Read more
impl<ST, DT> CastableFrom<ST, Initialized, Initialized> for DT
impl<ST, DT> CastableFrom<ST, Uninit, Uninit> for DT
Source§impl<T> CloneToUninit for Twhere
T: Clone,
impl<T> CloneToUninit for Twhere
T: Clone,
§impl<Q, K> Equivalent<K> for Q
impl<Q, K> Equivalent<K> for Q
§fn equivalent(&self, key: &K) -> bool
fn equivalent(&self, key: &K) -> bool
Checks if this value is equivalent to the given key. Read more
§impl<Q, K> Equivalent<K> for Q
impl<Q, K> Equivalent<K> for Q
§fn equivalent(&self, key: &K) -> bool
fn equivalent(&self, key: &K) -> bool
Compare self to
key and return true if they are equal.§impl<Q, K> Equivalent<K> for Q
impl<Q, K> Equivalent<K> for Q
§fn equivalent(&self, key: &K) -> bool
fn equivalent(&self, key: &K) -> bool
Checks if this value is equivalent to the given key. Read more
§impl<T> Instrument for T
impl<T> Instrument for T
§fn instrument(self, span: Span) -> Instrumented<Self> ⓘ
fn instrument(self, span: Span) -> Instrumented<Self> ⓘ
Source§impl<T> IntoEither for T
impl<T> IntoEither for T
Source§fn into_either(self, into_left: bool) -> Either<Self, Self> ⓘ
fn into_either(self, into_left: bool) -> Either<Self, Self> ⓘ
Converts
self into a Left variant of Either<Self, Self>
if into_left is true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read moreSource§fn into_either_with<F>(self, into_left: F) -> Either<Self, Self> ⓘ
fn into_either_with<F>(self, into_left: F) -> Either<Self, Self> ⓘ
Converts
self into a Left variant of Either<Self, Self>
if into_left(&self) returns true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read more