Struct TrunkNvmeReader
pub struct TrunkNvmeReader { /* private fields */ }Expand description
One encoded-weight source cursor. It owns no model payload; caller buffers bound each individual read/dequantization operation. A fixed 4 MiB read chunk amortizes syscalls across consecutive rows without holding model weights; it carries raw bytes only.
In mapped mode (open_mapped) the file is exposed through a read-only
memmap2 view so row access slices the OS page cache instead of issuing
syscalls. stats.reads then counts real syscalls (zero), while rows and
bytes still record the logical data volume touched.
Implementations§
§impl TrunkNvmeReader
impl TrunkNvmeReader
pub fn gemv_rows_into(
&mut self,
info: &TensorInfo,
first_row: usize,
rows: usize,
input: &[f32],
output: &mut [f32],
raw_scratch: &mut [u8],
row_scratch: &mut [f32],
) -> Result<(), TrunkNvmeError>
pub fn gemv_rows_into( &mut self, info: &TensorInfo, first_row: usize, rows: usize, input: &[f32], output: &mut [f32], raw_scratch: &mut [u8], row_scratch: &mut [f32], ) -> Result<(), TrunkNvmeError>
Execute a bounded streamed matrix-vector product over consecutive flattened rows. Each row is read, dequantized, dotted, then discarded.
pub fn gemv_expert_plane_into(
&mut self,
info: &TensorInfo,
expert: usize,
input: &[f32],
output: &mut [f32],
raw_scratch: &mut [u8],
row_scratch: &mut [f32],
) -> Result<usize, TrunkNvmeError>
pub fn gemv_expert_plane_into( &mut self, info: &TensorInfo, expert: usize, input: &[f32], output: &mut [f32], raw_scratch: &mut [u8], row_scratch: &mut [f32], ) -> Result<usize, TrunkNvmeError>
Execute one selected plane of an encoded [in, out, expert] tensor.
Only the selected expert plane is read; other expert weights remain
untouched on disk regardless of the model’s expert count or top-k.
§impl TrunkNvmeReader
impl TrunkNvmeReader
pub fn argmax_rows(
&mut self,
info: &TensorInfo,
input: &[f32],
raw_scratch: &mut [u8],
row_scratch: &mut [f32],
) -> Result<StreamedArgmax, TrunkNvmeError>
pub fn argmax_rows( &mut self, info: &TensorInfo, input: &[f32], raw_scratch: &mut [u8], row_scratch: &mut [f32], ) -> Result<StreamedArgmax, TrunkNvmeError>
Evaluate a row-addressable output projection without materialising a logits vector. The decoder scans one vocabulary row at a time and retains only the current winner, so a 248k-token vocabulary does not require a multi-megabyte hot-path allocation.
pub fn topk_rows(
&mut self,
info: &TensorInfo,
input: &[f32],
raw_scratch: &mut [u8],
row_scratch: &mut [f32],
out: &mut [StreamedArgmax],
) -> Result<(), TrunkNvmeError>
pub fn topk_rows( &mut self, info: &TensorInfo, input: &[f32], raw_scratch: &mut [u8], row_scratch: &mut [f32], out: &mut [StreamedArgmax], ) -> Result<(), TrunkNvmeError>
Stream the same projection retaining the out.len() highest logits,
sorted descending. Lower token IDs win exact ties, matching the
ascending row scan.
pub fn logits_and_ranks(
&mut self,
info: &TensorInfo,
input: &[f32],
target_rows: &[u32],
raw_scratch: &mut [u8],
row_scratch: &mut [f32],
out: &mut [(f32, u64)],
) -> Result<(), TrunkNvmeError>
pub fn logits_and_ranks( &mut self, info: &TensorInfo, input: &[f32], target_rows: &[u32], raw_scratch: &mut [u8], row_scratch: &mut [f32], out: &mut [(f32, u64)], ) -> Result<(), TrunkNvmeError>
Report each target_rows token’s logit and its rank across the whole
projection (rank 0 = argmax). Diagnostics for comparing against a
reference distribution: the target rows are read directly, then one
streamed pass counts how many vocabulary logits beat each target.
out[i] receives (logit, rank) for target_rows[i].
§impl TrunkNvmeReader
impl TrunkNvmeReader
pub fn open(
source: &Path,
tensor_data_start: u64,
) -> Result<TrunkNvmeReader, TrunkNvmeError>
pub fn open( source: &Path, tensor_data_start: u64, ) -> Result<TrunkNvmeReader, TrunkNvmeError>
Opens the resource.
pub fn open_mapped(
source: &Path,
tensor_data_start: u64,
) -> Result<TrunkNvmeReader, TrunkNvmeError>
pub fn open_mapped( source: &Path, tensor_data_start: u64, ) -> Result<TrunkNvmeReader, TrunkNvmeError>
Open the same source through a read-only OS mapping. Page residency is the OS’s decision; this reader still owns no decoded weights.
pub fn open_mapped_with_budget( source: &Path, tensor_data_start: u64, byte_budget: usize, ) -> Result<TrunkNvmeReader, TrunkNvmeError>
pub fn from_lease(
lease: Arc<ModelLease>,
tensor_data_start: u64,
mapped: bool,
scratch_budget: usize,
) -> Result<TrunkNvmeReader, TrunkNvmeError>
pub fn from_lease( lease: Arc<ModelLease>, tensor_data_start: u64, mapped: bool, scratch_budget: usize, ) -> Result<TrunkNvmeReader, TrunkNvmeError>
Consume the retained immutable source; no pathname is reopened.
pub fn mapped_bytes(&self) -> Option<&[u8]>
pub fn io_stats(&self) -> TrunkIoStats
pub fn io_stats(&self) -> TrunkIoStats
Cumulative physical I/O performed by this reader.
pub fn read_row_into(
&mut self,
info: &TensorInfo,
row: usize,
raw_scratch: &mut [u8],
out: &mut [f32],
) -> Result<usize, TrunkNvmeError>
pub fn read_row_into( &mut self, info: &TensorInfo, row: usize, raw_scratch: &mut [u8], out: &mut [f32], ) -> Result<usize, TrunkNvmeError>
Read and dequantize one flattened tensor row. For an expert tensor,
row includes its expert-plane offset; no entire expert tensor is read.
Auto Trait Implementations§
impl Freeze for TrunkNvmeReader
impl RefUnwindSafe for TrunkNvmeReader
impl Send for TrunkNvmeReader
impl Sync for TrunkNvmeReader
impl Unpin for TrunkNvmeReader
impl UnsafeUnpin for TrunkNvmeReader
impl UnwindSafe for TrunkNvmeReader
Blanket Implementations§
§impl<S, A> Aggregate<Result<S, Error>> for Awhere
A: Aggregate<S>,
impl<S, A> Aggregate<Result<S, Error>> for Awhere
A: Aggregate<S>,
Source§impl<T> BorrowMut<T> for Twhere
T: ?Sized,
impl<T> BorrowMut<T> for Twhere
T: ?Sized,
Source§fn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
impl<ST, DT> CastableFrom<ST, Initialized, Initialized> for DT
impl<ST, DT> CastableFrom<ST, Uninit, Uninit> for DT
§impl<T> Instrument for T
impl<T> Instrument for T
§fn instrument(self, span: Span) -> Instrumented<Self> ⓘ
fn instrument(self, span: Span) -> Instrumented<Self> ⓘ
Source§impl<T> IntoEither for T
impl<T> IntoEither for T
Source§fn into_either(self, into_left: bool) -> Either<Self, Self> ⓘ
fn into_either(self, into_left: bool) -> Either<Self, Self> ⓘ
self into a Left variant of Either<Self, Self>
if into_left is true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read moreSource§fn into_either_with<F>(self, into_left: F) -> Either<Self, Self> ⓘ
fn into_either_with<F>(self, into_left: F) -> Either<Self, Self> ⓘ
self into a Left variant of Either<Self, Self>
if into_left(&self) returns true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read more