pub struct BoundedInferenceCache { /* private fields */ }Expand description
Bounded in-memory KV cache with capacity limits and deterministic LRU eviction.
Implementations§
Source§impl BoundedInferenceCache
impl BoundedInferenceCache
Sourcepub fn new(max_entries: usize, max_bytes: usize) -> Self
pub fn new(max_entries: usize, max_bytes: usize) -> Self
Creates a default BoundedInferenceCache.
Sourcepub fn current_bytes(&self) -> usize
pub fn current_bytes(&self) -> usize
Total bytes consumed by stored KV cache float buffers.
Sourcepub fn restore_if_match<R, F: FnOnce(&[f32]) -> R>(
&mut self,
key: &InferenceCacheKey,
f: F,
) -> Option<R>
pub fn restore_if_match<R, F: FnOnce(&[f32]) -> R>( &mut self, key: &InferenceCacheKey, f: F, ) -> Option<R>
Restore cached KV slice via a closure without cloning if the key matches and is complete.
Sourcepub fn restore_if_match_exact<R, F: FnOnce(&[f32]) -> R>(
&mut self,
key: &InferenceCacheKey,
exact_prefix: &[u32],
f: F,
) -> Option<R>
pub fn restore_if_match_exact<R, F: FnOnce(&[f32]) -> R>( &mut self, key: &InferenceCacheKey, exact_prefix: &[u32], f: F, ) -> Option<R>
Restore cached KV slice verifying exact token prefix match to eliminate any hash collision risk.
Sourcepub fn insert(
&mut self,
key: InferenceCacheKey,
data: Box<[f32]>,
token_count: u32,
is_complete: bool,
) -> Result<(), &'static str>
pub fn insert( &mut self, key: InferenceCacheKey, data: Box<[f32]>, token_count: u32, is_complete: bool, ) -> Result<(), &'static str>
Insert completed prefill KV data without token prefix slice.
Auto Trait Implementations§
impl Freeze for BoundedInferenceCache
impl RefUnwindSafe for BoundedInferenceCache
impl Send for BoundedInferenceCache
impl Sync for BoundedInferenceCache
impl Unpin for BoundedInferenceCache
impl UnsafeUnpin for BoundedInferenceCache
impl UnwindSafe for BoundedInferenceCache
Blanket Implementations§
§impl<S, A> Aggregate<Result<S, Error>> for Awhere
A: Aggregate<S>,
impl<S, A> Aggregate<Result<S, Error>> for Awhere
A: Aggregate<S>,
Aggregate shares in an MPC protocol.
Source§impl<T> BorrowMut<T> for Twhere
T: ?Sized,
impl<T> BorrowMut<T> for Twhere
T: ?Sized,
Source§fn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
Mutably borrows from an owned value. Read more
impl<ST, DT> CastableFrom<ST, Initialized, Initialized> for DT
impl<ST, DT> CastableFrom<ST, Uninit, Uninit> for DT
§impl<T> Instrument for T
impl<T> Instrument for T
§fn instrument(self, span: Span) -> Instrumented<Self> ⓘ
fn instrument(self, span: Span) -> Instrumented<Self> ⓘ
Source§impl<T> IntoEither for T
impl<T> IntoEither for T
Source§fn into_either(self, into_left: bool) -> Either<Self, Self> ⓘ
fn into_either(self, into_left: bool) -> Either<Self, Self> ⓘ
Converts
self into a Left variant of Either<Self, Self>
if into_left is true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read moreSource§fn into_either_with<F>(self, into_left: F) -> Either<Self, Self> ⓘ
fn into_either_with<F>(self, into_left: F) -> Either<Self, Self> ⓘ
Converts
self into a Left variant of Either<Self, Self>
if into_left(&self) returns true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read more