Struct ModelMemoryProfile
pub struct ModelMemoryProfile {
pub weight_bytes: u64,
pub scratch_activation_bytes: u64,
pub device_context_overhead_bytes: u64,
pub graph_capture_headroom_bytes: u64,
}Expand description
Static memory profile of a loaded model on a device backend.
Fields§
§weight_bytes: u64Bytes occupied by static model weights and quantization tables.
scratch_activation_bytes: u64Working memory for intermediate activation tensors during decode/prefill.
device_context_overhead_bytes: u64Fixed driver / runtime context overhead (e.g. CUDA context, DirectML device).
graph_capture_headroom_bytes: u64Headroom reserved for static execution graphs (CUDA graphs / DirectML graphs).
Implementations§
§impl ModelMemoryProfile
impl ModelMemoryProfile
pub fn new(
weight_bytes: u64,
scratch_activation_bytes: u64,
device_context_overhead_bytes: u64,
graph_capture_headroom_bytes: u64,
) -> Result<ModelMemoryProfile, BudgetError>
pub fn new( weight_bytes: u64, scratch_activation_bytes: u64, device_context_overhead_bytes: u64, graph_capture_headroom_bytes: u64, ) -> Result<ModelMemoryProfile, BudgetError>
Creates ModelMemoryProfile with the given state.
pub fn static_device_bytes(&self) -> u64
pub fn static_device_bytes(&self) -> u64
Total bytes required by the static model runtime before dynamic KV pools.
Trait Implementations§
§impl Clone for ModelMemoryProfile
impl Clone for ModelMemoryProfile
§fn clone(&self) -> ModelMemoryProfile
fn clone(&self) -> ModelMemoryProfile
Returns a duplicate of the value. Read more
1.0.0 (const: unstable) · Source§fn clone_from(&mut self, source: &Self)
fn clone_from(&mut self, source: &Self)
Performs copy-assignment from
source. Read moreimpl Copy for ModelMemoryProfile
§impl Debug for ModelMemoryProfile
impl Debug for ModelMemoryProfile
impl Eq for ModelMemoryProfile
§impl PartialEq for ModelMemoryProfile
impl PartialEq for ModelMemoryProfile
impl StructuralPartialEq for ModelMemoryProfile
Auto Trait Implementations§
impl Freeze for ModelMemoryProfile
impl RefUnwindSafe for ModelMemoryProfile
impl Send for ModelMemoryProfile
impl Sync for ModelMemoryProfile
impl Unpin for ModelMemoryProfile
impl UnsafeUnpin for ModelMemoryProfile
impl UnwindSafe for ModelMemoryProfile
Blanket Implementations§
§impl<S, A> Aggregate<Result<S, Error>> for Awhere
A: Aggregate<S>,
impl<S, A> Aggregate<Result<S, Error>> for Awhere
A: Aggregate<S>,
Aggregate shares in an MPC protocol.
Source§impl<T> BorrowMut<T> for Twhere
T: ?Sized,
impl<T> BorrowMut<T> for Twhere
T: ?Sized,
Source§fn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
Mutably borrows from an owned value. Read more
impl<ST, DT> CastableFrom<ST, Initialized, Initialized> for DT
impl<ST, DT> CastableFrom<ST, Uninit, Uninit> for DT
Source§impl<T> CloneToUninit for Twhere
T: Clone,
impl<T> CloneToUninit for Twhere
T: Clone,
§impl<Q, K> Equivalent<K> for Q
impl<Q, K> Equivalent<K> for Q
§fn equivalent(&self, key: &K) -> bool
fn equivalent(&self, key: &K) -> bool
Checks if this value is equivalent to the given key. Read more
§impl<Q, K> Equivalent<K> for Q
impl<Q, K> Equivalent<K> for Q
§fn equivalent(&self, key: &K) -> bool
fn equivalent(&self, key: &K) -> bool
Compare self to
key and return true if they are equal.§impl<Q, K> Equivalent<K> for Q
impl<Q, K> Equivalent<K> for Q
§fn equivalent(&self, key: &K) -> bool
fn equivalent(&self, key: &K) -> bool
Checks if this value is equivalent to the given key. Read more
§impl<T> Instrument for T
impl<T> Instrument for T
§fn instrument(self, span: Span) -> Instrumented<Self> ⓘ
fn instrument(self, span: Span) -> Instrumented<Self> ⓘ
Source§impl<T> IntoEither for T
impl<T> IntoEither for T
Source§fn into_either(self, into_left: bool) -> Either<Self, Self> ⓘ
fn into_either(self, into_left: bool) -> Either<Self, Self> ⓘ
Converts
self into a Left variant of Either<Self, Self>
if into_left is true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read moreSource§fn into_either_with<F>(self, into_left: F) -> Either<Self, Self> ⓘ
fn into_either_with<F>(self, into_left: F) -> Either<Self, Self> ⓘ
Converts
self into a Left variant of Either<Self, Self>
if into_left(&self) returns true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read more