Struct QTensorEngine 

pub struct QTensorEngine {
    pub pipeline: ComputePipeline,
    pub embedding_pipeline: ComputePipeline,
    pub is_initialized: bool,
    pub gguf_mmap: Option<Arc<Mmap>>,
    pub p64_index: Option<P64TensorIndex>,
    pub tensor_index_cache: Option<GgufTensorIndex>,
    pub tensor_data_offset: u64,
    pub hyperparams: GgufHyperparams,
    pub max_tensor_bytes: usize,
    /* private fields */
}

Fields

pipeline: ComputePipeline

embedding_pipeline: ComputePipeline

GPU-side Q6_K embedding dequant + matmul (zero CPU dequant).

is_initialized: bool

gguf_mmap: Option<Arc<Mmap>>

Memory-mapped GGUF file (set after load_gguf).

p64_index: Option<P64TensorIndex>

Cached P64 index after adopt_resident_p64_* — decode must not re-CRC the container.

tensor_index_cache: Option<GgufTensorIndex>

Cached synthetic GGUF index built from p64_index (or from GGUF parse).

tensor_data_offset: u64

Byte offset into the mmap where tensor data begins.

hyperparams: GgufHyperparams

max_tensor_bytes: usize

Implementations

impl QTensorEngine

pub fn dispatch_output_logits_into( &self, index: &GgufTensorIndex, hidden: &[f32], emb_dim: usize, logits_out: &mut [f32], ) -> usize

Final logits via chunked projection into logits_out (fills min(vocab, buf) rows).

pub fn decode_lexicon_bound( &self, _logits: &[f32], valid_lexicon_ids: &[u64], ) -> u64

impl QTensorEngine

pub fn cuda_prepared_telemetry(&self) -> Option<CudaPreparedTelemetry>

impl QTensorEngine

pub fn dispatch_quantized_token_embedding( &self, raw_embd: &[u8], ggml_type: u32, n_embd: u32, weight_tensor: &QTensor, ) -> Option<Vec<f32>>

Upload raw quantized embedding bytes to the GPU and matmul without CPU dequant. Returns None when the GGML type has no WGSL kernel (caller uses CPU fallback).

pub fn dispatch_fused_transformer_block( &self, tensor: &QTensor, input_activations: &[f32], ) -> Vec<f32>

impl QTensorEngine

pub fn dispatch_prefill_chunk( &mut self, index: &GgufTensorIndex, batch_hidden: &mut [f32], emb_dim: usize, n_tokens: u32, batch_start_token_idx: u32, scratch_a: &mut [f32], scratch_b: &mut [f32], max_layers: u32, ) -> bool

Chunked prefill: populate KV arena for n_tokens prompt positions starting at batch_start.

pub fn dispatch_transformer_layer( &mut self, index: &GgufTensorIndex, layer: u32, token_idx: u32, hidden: &mut [f32], emb_dim: usize, scratch_a: &mut [f32], scratch_b: &mut [f32], ) -> bool

One transformer block using real mmap tensor offsets (stack buffers only).

pub fn try_cuda_mega_pass_decode( &mut self, index: &GgufTensorIndex, hidden: &mut [f32], emb_dim: usize, token_idx: u32, ) -> Option<u32>

Attempt the CUDA mega-pass: all layers in one fenced CUDA stream. Returns Some(token_id) on success, None to fall back to per-layer path.

pub fn try_cuda_mega_pass_decode_token( &mut self, index: &GgufTensorIndex, token_id: u32, hidden: &mut [f32], emb_dim: usize, token_idx: u32, ) -> Option<u32>

Decode directly from a token id using the resident Q8 embedding table.

This path avoids CPU embedding dequantization and the full hidden-state H2D upload.

pub fn dispatch_transformer_forward( &mut self, index: &GgufTensorIndex, hidden: &mut [f32], emb_dim: usize, scratch_a: &mut [f32], scratch_b: &mut [f32], token_idx: u32, max_layers: u32, ) -> u32

Sequential layer-by-layer forward (one tensor payload in VRAM at a time). max_layers: 0 runs all blocks; otherwise caps how many layers execute.

pub fn verify_topology_draft_batch( &mut self, index: &GgufTensorIndex, ctx: &mut Vec<u32>, draft: &TopologyDraftBatch, emb_dim: usize, emb_buf: &mut [f32], scratch_a: &mut [f32], scratch_b: &mut [f32], max_layers: u32, max_vocab_chunks: u32, ) -> u32

Topological speculative verify — accept longest draft prefix (B3.1d).

impl QTensorEngine

pub fn dispatch_gemm_into( &self, index: &GgufTensorIndex, info: &GgufTensorInfo, input: &[f32], out: &mut [f32], n_in: usize, n_out: usize, ) -> bool

Quantized GEMM into caller out using reused GPU buffers (Q6_K) or CPU dequant fallback.

impl QTensorEngine

pub fn device(&self) -> &Device

Shared process-wide wgpu device (LLM + render coexistence).

pub fn queue(&self) -> &Queue

Shared process-wide wgpu queue.

pub async fn try_new() -> Result<Self, String>

pub fn new() -> Self

pub fn reset_kv_cache(&mut self)

Zero the static KV arena at the start of a new decode context (zero heap in decode).

pub fn get_kv_cache_cpu(&self) -> Option<&[f32]>

pub fn set_kv_cache_cpu(&mut self, data: &[f32])

pub fn read_kv_cache_gpu(&self) -> Option<Vec<f32>>

Read the entire GPU KV-cache arena back to host as f32 (native, cold path — forge KV capture, not the decode hot path). The returned flat buffer is interpretable via KvCacheLayout:: k_index/v_index only when the layout is f32 (int8 KV disabled at load); with an int8 layout the bytes are packed i8 lanes + scales and this returns None.

pub fn capture_kv_f32( &self, n_tokens: u32, max_per_layer: usize, ) -> Option<KvCapture>

Decode the current f32 KV cache into per-layer K and V vectors for token positions 0..n_tokens (capped at max_per_layer per layer per stream). The GPU-readback capture route for the sparse-KV-dictionary go/no-go — reads the real decode-path K/V straight from VRAM, no CPU reference forward. Returns None on an int8 layout or if readback fails.

impl QTensorEngine

pub fn kv_cache_bytes(&self) -> u64

pub fn load_gguf_checked( &mut self, path: &str, ) -> Result<GgufLoadReport, String>

pub fn load_gguf(&mut self, path: &str)

Memory-map a GGUF file so tensor bytes are accessible without heap allocation. Call this once after new(), before the first dispatch_fused_transformer_block.

pub fn load_model_checked( &mut self, path: &str, ) -> Result<GgufLoadReport, String>

Memory-map and auto-detect a supported local model container.

Canonical P64 is detected by its exact p64\0 magic and adopted through the P64 validation path. All other inputs are passed to the GGUF parser, which rejects malformed or unsupported data.

pub fn load_model(&mut self, path: &str)

Fail-soft wrapper retained for the agent decode path.

pub fn adopt_resident_p64_mmap( &mut self, mmap: Arc<Mmap>, ) -> Result<GgufLoadReport, String>

Boot from an already-mapped P64 weight container (native). Mirrors the GGUF adopt_resident_mmap but for the P64 format: validates + builds a synthetic GGUF index from the manifest, points the byte source at the P64 bytes (tensor_data_start = 0, absolute blob offsets), reserves the GEMM/KV arenas, makes the (verbatim) output projection resident, and builds the resident 2-bit ternary-FFN dispatcher from the FFN blobs. The attention/norm/embed tensors stay at source precision and run the standard GGUF hot path.

pub fn adopt_resident_q42_mmap( &mut self, mmap: Arc<Mmap>, ) -> Result<GgufLoadReport, String>

👎Deprecated:

use adopt_resident_p64_mmap

Compatibility alias for the historical pre-P64 API name.

pub fn ternary_ffn_resident_len(&self) -> usize

A1b: number of resident ternary FFN tensors (0 unless a ternary P64 was adopted). Lets a test confirm the GPU resident path is actually populated (not a silent CPU-only fallback).

pub fn adopt_resident_mmap( &mut self, mmap: Arc<Mmap>, ) -> Result<GgufLoadReport, String>

Attach an already-mapped resident GGUF (shared with orchestrator slot).

pub fn bench_empty_submit_roundtrip(&self, n: u32) -> u64

Decode-profiler: wall-clock for n EMPTY submit → poll(Maintain::Wait) round-trips (no compute dispatched). Isolates the fixed CPU↔GPU fence latency: if a token’s forward time ≈ (its round-trip count × this per-round-trip cost), the bottleneck is synchronization, not math; if forward ≫ that, the kernels themselves are slow. Does NOT touch GPU_WAIT_COUNT.

impl QTensorEngine

pub fn dispatch_output_argmax_chunked( &self, index: &GgufTensorIndex, hidden: &[f32], emb_dim: usize, chunk_logits: &mut [f32], max_chunks: u32, sieve_mask: Option<&SieveStateMask>, ) -> Option<StreamingArgmaxResult>

Chunked vocabulary projection with streaming argmax (zero heap, stack chunk buffer only). max_chunks: 0 sweeps the full vocabulary; otherwise caps chunk iterations (tests).

pub fn dispatch_output_top1_chunked( &self, index: &GgufTensorIndex, hidden: &[f32], emb_dim: usize, ) -> Option<StreamingArgmaxResult>

Native decode fast path: output projection plus GPU block argmax (k=1) with one tiny candidate readback after all vocab chunks have been submitted. This avoids the full chunk-logit readback in Self::dispatch_output_argmax_chunked and avoids heap allocation in the decode loop.

pub fn dispatch_output_topk_chunked( &self, index: &GgufTensorIndex, hidden: &[f32], emb_dim: usize, k: usize, ) -> Option<Vec<TopKItem>>

A1a: GPU top-k over the output projection — the logits stay on-GPU (gemm_output_buf), the top-k kernel reduces them per chunk, and only K (id, logit) candidates are read back (vs the 196 KB/token full-logit readback + CPU argmax in dispatch_output_argmax_chunked). Returns the merged global top-K, or None to signal the caller to fall back to the argmax path. v1: no sieve coupling (caller routes here only when no sieve mask is active).

pub fn apply_output_norm_inplace( &self, index: &GgufTensorIndex, hidden: &mut [f32], emb_dim: usize, ) -> bool

Final output_norm RMSNorm in-place before vocabulary projection (Pre-Norm LLM tail). REQUIRED on all targets — native previously skipped it → logits from an un-normed hidden.

impl QTensorEngine

pub fn resident_dispatches_per_token(&self) -> Option<u32>

Exact steady-state compute dispatch count for the prepared resident plan.

pub fn resident_readback_bytes_per_token(&self) -> Option<u32>

Exact candidate staging copy size for one resident greedy token.

pub fn dispatch_token_forward_resident( &mut self, index: &GgufTensorIndex, emb: &[f32], token_idx: u32, ) -> Option<StreamingArgmaxResult>

Single-fence resident-token decode (greedy top-1). Returns the argmax token, or None on any ineligibility (caller falls back to the legacy per-layer path).

pub fn dispatch_token_forward_resident_hidden( &mut self, index: &GgufTensorIndex, emb: &[f32], token_idx: u32, out_hidden: &mut [f32], ) -> bool

Sampler-compatible resident forward: same single-fence layer stack + output RMSNorm, then read back the normed hidden into out_hidden so the caller can project full logits + sample on CPU without the legacy ~107-fence path.

emb is the token embedding input; out_hidden receives post-output-norm state (may be the same logical buffer only if the caller copies input first — they must not alias while the upload of emb is live; pass distinct slices).

impl QTensorEngine

pub fn gemm_parity_probe( &mut self, info: &GgufTensorInfo, raw: &[u8], input: &[f32], gpu_out: &mut [f32], cpu_out: &mut [f32], n_in: usize, n_out: usize, ) -> bool

W3 kernel-parity probe (test/diagnostic): run the GPU GEMM (dispatch_gemm_raw_into) and the CPU reference (stack_gemm_quant) on the SAME quantized weights + input, writing each into a caller-provided buffer. Ensures the GEMM buffers exist first, so a fresh engine (no model loaded) can be probed directly. Returns true only if both ran; the caller compares the two outputs with crate::llm_kernel_parity. Enable crate::llm_gpu_profiler around the call to witness that the GPU path actually executed rather than silently falling back to the CPU.