Struct BlockGeometry
pub struct BlockGeometry {
pub tokens_per_block: u32,
pub bytes_per_block: u32,
}Expand description
Geometry of a physical KV cache block/page.
Fields§
§tokens_per_block: u32Number of tokens packed per physical block (e.g. 16, 32).
bytes_per_block: u32Byte size of a single physical block (layers * heads * dim * elem_size).
Implementations§
§impl BlockGeometry
impl BlockGeometry
pub fn new(
tokens_per_block: u32,
bytes_per_block: u32,
) -> Result<BlockGeometry, BudgetError>
pub fn new( tokens_per_block: u32, bytes_per_block: u32, ) -> Result<BlockGeometry, BudgetError>
Creates BlockGeometry with the given state.
pub fn blocks_for_tokens(&self, tokens: u32) -> Result<u32, BudgetError>
pub fn blocks_for_tokens(&self, tokens: u32) -> Result<u32, BudgetError>
Calculate the minimum number of blocks needed to house tokens,
accounting for page boundary ceiling rounding (fragmentation).
pub fn bytes_for_blocks(&self, blocks: u32) -> Result<u64, BudgetError>
pub fn bytes_for_blocks(&self, blocks: u32) -> Result<u64, BudgetError>
Calculate the total bytes occupied by blocks.
Trait Implementations§
§impl Clone for BlockGeometry
impl Clone for BlockGeometry
§fn clone(&self) -> BlockGeometry
fn clone(&self) -> BlockGeometry
Returns a duplicate of the value. Read more
1.0.0 (const: unstable) · Source§fn clone_from(&mut self, source: &Self)
fn clone_from(&mut self, source: &Self)
Performs copy-assignment from
source. Read moreimpl Copy for BlockGeometry
§impl Debug for BlockGeometry
impl Debug for BlockGeometry
impl Eq for BlockGeometry
§impl PartialEq for BlockGeometry
impl PartialEq for BlockGeometry
impl StructuralPartialEq for BlockGeometry
Auto Trait Implementations§
impl Freeze for BlockGeometry
impl RefUnwindSafe for BlockGeometry
impl Send for BlockGeometry
impl Sync for BlockGeometry
impl Unpin for BlockGeometry
impl UnsafeUnpin for BlockGeometry
impl UnwindSafe for BlockGeometry
Blanket Implementations§
§impl<S, A> Aggregate<Result<S, Error>> for Awhere
A: Aggregate<S>,
impl<S, A> Aggregate<Result<S, Error>> for Awhere
A: Aggregate<S>,
Aggregate shares in an MPC protocol.
Source§impl<T> BorrowMut<T> for Twhere
T: ?Sized,
impl<T> BorrowMut<T> for Twhere
T: ?Sized,
Source§fn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
Mutably borrows from an owned value. Read more
impl<ST, DT> CastableFrom<ST, Initialized, Initialized> for DT
impl<ST, DT> CastableFrom<ST, Uninit, Uninit> for DT
Source§impl<T> CloneToUninit for Twhere
T: Clone,
impl<T> CloneToUninit for Twhere
T: Clone,
§impl<Q, K> Equivalent<K> for Q
impl<Q, K> Equivalent<K> for Q
§fn equivalent(&self, key: &K) -> bool
fn equivalent(&self, key: &K) -> bool
Checks if this value is equivalent to the given key. Read more
§impl<Q, K> Equivalent<K> for Q
impl<Q, K> Equivalent<K> for Q
§fn equivalent(&self, key: &K) -> bool
fn equivalent(&self, key: &K) -> bool
Compare self to
key and return true if they are equal.§impl<Q, K> Equivalent<K> for Q
impl<Q, K> Equivalent<K> for Q
§fn equivalent(&self, key: &K) -> bool
fn equivalent(&self, key: &K) -> bool
Checks if this value is equivalent to the given key. Read more
§impl<T> Instrument for T
impl<T> Instrument for T
§fn instrument(self, span: Span) -> Instrumented<Self> ⓘ
fn instrument(self, span: Span) -> Instrumented<Self> ⓘ
Source§impl<T> IntoEither for T
impl<T> IntoEither for T
Source§fn into_either(self, into_left: bool) -> Either<Self, Self> ⓘ
fn into_either(self, into_left: bool) -> Either<Self, Self> ⓘ
Converts
self into a Left variant of Either<Self, Self>
if into_left is true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read moreSource§fn into_either_with<F>(self, into_left: F) -> Either<Self, Self> ⓘ
fn into_either_with<F>(self, into_left: F) -> Either<Self, Self> ⓘ
Converts
self into a Left variant of Either<Self, Self>
if into_left(&self) returns true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read more