Skip to main content

TrunkNvmeReader

Struct TrunkNvmeReader 

pub struct TrunkNvmeReader { /* private fields */ }
Expand description

One encoded-weight source cursor. It owns no model payload; caller buffers bound each individual read/dequantization operation. A fixed 4 MiB read chunk amortizes syscalls across consecutive rows without holding model weights; it carries raw bytes only.

In mapped mode (open_mapped) the file is exposed through a read-only memmap2 view so row access slices the OS page cache instead of issuing syscalls. stats.reads then counts real syscalls (zero), while rows and bytes still record the logical data volume touched.

Implementations§

§

impl TrunkNvmeReader

pub fn gemv_rows_into( &mut self, info: &TensorInfo, first_row: usize, rows: usize, input: &[f32], output: &mut [f32], raw_scratch: &mut [u8], row_scratch: &mut [f32], ) -> Result<(), TrunkNvmeError>

Execute a bounded streamed matrix-vector product over consecutive flattened rows. Each row is read, dequantized, dotted, then discarded.

pub fn gemv_expert_plane_into( &mut self, info: &TensorInfo, expert: usize, input: &[f32], output: &mut [f32], raw_scratch: &mut [u8], row_scratch: &mut [f32], ) -> Result<usize, TrunkNvmeError>

Execute one selected plane of an encoded [in, out, expert] tensor. Only the selected expert plane is read; other expert weights remain untouched on disk regardless of the model’s expert count or top-k.

§

impl TrunkNvmeReader

pub fn argmax_rows( &mut self, info: &TensorInfo, input: &[f32], raw_scratch: &mut [u8], row_scratch: &mut [f32], ) -> Result<StreamedArgmax, TrunkNvmeError>

Evaluate a row-addressable output projection without materialising a logits vector. The decoder scans one vocabulary row at a time and retains only the current winner, so a 248k-token vocabulary does not require a multi-megabyte hot-path allocation.

pub fn topk_rows( &mut self, info: &TensorInfo, input: &[f32], raw_scratch: &mut [u8], row_scratch: &mut [f32], out: &mut [StreamedArgmax], ) -> Result<(), TrunkNvmeError>

Stream the same projection retaining the out.len() highest logits, sorted descending. Lower token IDs win exact ties, matching the ascending row scan.

pub fn logits_and_ranks( &mut self, info: &TensorInfo, input: &[f32], target_rows: &[u32], raw_scratch: &mut [u8], row_scratch: &mut [f32], out: &mut [(f32, u64)], ) -> Result<(), TrunkNvmeError>

Report each target_rows token’s logit and its rank across the whole projection (rank 0 = argmax). Diagnostics for comparing against a reference distribution: the target rows are read directly, then one streamed pass counts how many vocabulary logits beat each target. out[i] receives (logit, rank) for target_rows[i].

§

impl TrunkNvmeReader

pub fn open( source: &Path, tensor_data_start: u64, ) -> Result<TrunkNvmeReader, TrunkNvmeError>

Opens the resource.

pub fn open_mapped( source: &Path, tensor_data_start: u64, ) -> Result<TrunkNvmeReader, TrunkNvmeError>

Open the same source through a read-only OS mapping. Page residency is the OS’s decision; this reader still owns no decoded weights.

pub fn open_mapped_with_budget( source: &Path, tensor_data_start: u64, byte_budget: usize, ) -> Result<TrunkNvmeReader, TrunkNvmeError>

pub fn from_lease( lease: Arc<ModelLease>, tensor_data_start: u64, mapped: bool, scratch_budget: usize, ) -> Result<TrunkNvmeReader, TrunkNvmeError>

Consume the retained immutable source; no pathname is reopened.

pub fn is_mapped(&self) -> bool

Whether row access is served from the page-cache mapping.

pub fn mapped_bytes(&self) -> Option<&[u8]>

pub fn io_stats(&self) -> TrunkIoStats

Cumulative physical I/O performed by this reader.

pub fn read_row_into( &mut self, info: &TensorInfo, row: usize, raw_scratch: &mut [u8], out: &mut [f32], ) -> Result<usize, TrunkNvmeError>

Read and dequantize one flattened tensor row. For an expert tensor, row includes its expert-plane offset; no entire expert tensor is read.

Auto Trait Implementations§

Blanket Implementations§

§

impl<S, A> Aggregate<Result<S, Error>> for A
where A: Aggregate<S>,

§

fn from_shares<T>(iter: T) -> Result<A, Error>
where T: IntoIterator<Item = Result<S, Error>>,

Aggregate shares in an MPC protocol.
Source§

impl<T> Any for T
where T: 'static + ?Sized,

Source§

fn type_id(&self) -> TypeId

Gets the TypeId of self. Read more
Source§

impl<T> Borrow<T> for T
where T: ?Sized,

Source§

fn borrow(&self) -> &T

Immutably borrows from an owned value. Read more
Source§

impl<T> BorrowMut<T> for T
where T: ?Sized,

Source§

fn borrow_mut(&mut self) -> &mut T

Mutably borrows from an owned value. Read more
§

impl<ST, DT> CastableFrom<ST, Initialized, Initialized> for DT
where ST: ?Sized, DT: ?Sized,

§

impl<ST, DT> CastableFrom<ST, Uninit, Uninit> for DT
where ST: ?Sized, DT: ?Sized,

§

impl<T> Downcast<T> for T

§

fn downcast(&self) -> &T

Source§

impl<T> From<T> for T

Source§

fn from(t: T) -> T

Returns the argument unchanged.

§

impl<T> Instrument for T

§

fn instrument(self, span: Span) -> Instrumented<Self> ⓘ

Instruments this type with the provided [Span], returning an Instrumented wrapper. Read more
§

fn in_current_span(self) -> Instrumented<Self> ⓘ

Instruments this type with the current Span, returning an Instrumented wrapper. Read more
Source§

impl<T, U> Into<U> for T
where U: From<T>,

Source§

fn into(self) -> U

Calls U::from(self).

That is, this conversion is whatever the implementation of From<T> for U chooses to do.

Source§

impl<T> IntoEither for T

Source§

fn into_either(self, into_left: bool) -> Either<Self, Self> ⓘ

Converts self into a Left variant of Either<Self, Self> if into_left is true. Converts self into a Right variant of Either<Self, Self> otherwise. Read more
Source§

fn into_either_with<F>(self, into_left: F) -> Either<Self, Self> ⓘ
where F: FnOnce(&Self) -> bool,

Converts self into a Left variant of Either<Self, Self> if into_left(&self) returns true. Converts self into a Right variant of Either<Self, Self> otherwise. Read more
§

impl<T> Pointable for T

§

const ALIGN: usize

The alignment of pointer.
§

type Init = T

The type for initializers.
§

unsafe fn init(init: <T as Pointable>::Init) -> usize

Initializes a with the given initializer. Read more
§

unsafe fn deref<'a>(ptr: usize) -> &'a T

Dereferences the given pointer. Read more
§

unsafe fn deref_mut<'a>(ptr: usize) -> &'a mut T

Mutably dereferences the given pointer. Read more
§

unsafe fn drop(ptr: usize)

Drops the object pointed to by the given pointer. Read more
§

impl<T> PolicyExt for T
where T: ?Sized,

§

fn and<P, B, E>(self, other: P) -> And<T, P>
where T: Sized + Policy<B, E>, P: Policy<B, E>,

Create a new Policy that returns [Action::Follow] only if self and other return Action::Follow. Read more
§

fn or<P, B, E>(self, other: P) -> Or<T, P>
where T: Sized + Policy<B, E>, P: Policy<B, E>,

Create a new Policy that returns [Action::Follow] if either self or other returns Action::Follow. Read more
§

impl<T> Read<Exclusive, BecauseExclusive> for T
where T: ?Sized,

Source§

impl<T> Same for T

Source§

type Output = T

Should always be Self
Source§

impl<T, U> TryFrom<U> for T
where U: Into<T>,

Source§

type Error = !

The type returned in the event of a conversion error.
Source§

fn try_from(value: U) -> Result<T, !>

Performs the conversion.
Source§

impl<T, U> TryInto<U> for T
where U: TryFrom<T>,

Source§

type Error = <U as TryFrom<T>>::Error

The type returned in the event of a conversion error.
Source§

fn try_into(self) -> Result<U, <U as TryFrom<T>>::Error>

Performs the conversion.
§

impl<T> Upcast<T> for T

§

fn upcast(&self) -> Option<&T>

§

impl<V, T> VZip<V> for T
where V: MultiLane<T>,

§

fn vzip(self) -> V

§

impl<T> WasmNotSend for T
where T: Send,

§

impl<T> WasmNotSendSync for T
where T: WasmNotSend + WasmNotSync,

§

impl<T> WasmNotSync for T
where T: Sync,

§

impl<T> WithSubscriber for T

§

fn with_subscriber<S>(self, subscriber: S) -> WithDispatch<Self> ⓘ
where S: Into<Dispatch>,

Attaches the provided Subscriber to this type, returning a [WithDispatch] wrapper. Read more
§

fn with_current_subscriber(self) -> WithDispatch<Self> ⓘ

Attaches the current default Subscriber to this type, returning a [WithDispatch] wrapper. Read more