Module cuda_lane 

CUDA inference lane — dense batch GEMM via persistent WMMA (mode=cuda).

Goal

Prefer tensor-core dense matmul for batch prefill-shaped GEMMs when:

• InferenceMode::CudaTc is active (prefer_tensor_core_gemm)
• dims can be padded to multiples of 16
• a dense f32 weight matrix is available (f16 p64 expand, or one-shot dequant)

Weight rows are cached on the CUDA slab by content fingerprint so subsequent chunks do not re-upload the same matrix (no host thrash for repeated layers).

Honest limits

• Not a fused Q4_K dequant-GEMV on-device (llama.cpp-class) — that remains M2b.
• Quantized weights must be dequantized once to dense f32 before first cache insert.
• Slab is finite (256 MiB); LRU-ish eviction of oldest entries when full.

Structs

MegaPassLayerDims

Per-layer matmul dimensions resolved during cold plan construction.

MegaPassLayerWeights

Per-layer weight references for the prepared CUDA decode pass.

Enums

MegaPassWeightLayout

Constants

MAX_DENSE_ELEMS

Max f32 elements per densified matrix (~192 MiB) — covers 3B FFN (~25M) with headroom.

Q8_0_BLOCK_BYTES

Q8_0_BLOCK_ELEMS

Traits

MegaPassPlanView

Immutable view of the cold-prepared weights and tensor metadata.

Functions

cache_dense_weight

Insert or refresh a dense weight matrix (row-major n_out × n_in).

cache_dense_weight_direct

Allocate a 2 MiB-aligned buffer and fill it via fill — skips the intermediate Vec<f32> that cache_dense_weight requires. The closure receives a row-major n_out × n_in slice to write into directly.

clear_weight_cache

Clear weight cache (tests / model swap).

dense_weight_cached

True if a dense weight for key is already in the host TC cache.

device_kv_ready

True when a permanent device KV arena is resident (P4 path eligible).

ensure_device_kv_cache

Reserve permanent device KV matching host KvCacheLayout f32 indices (P4). Call before heavy weight preload when possible so the slab still has room. Returns false if layout is invalid or the slab cannot hold the arena.

preload_q4k_soa_weights

Bulk-preload Q4_K_SOA weight blobs into the multi-weight CUDA slab. Call once at plan build so first-token decode does not thrash PCIe. Each entry is (raw_bytes, n_in, n_out). Returns how many newly resident.

preload_resident_blob

Upload an immutable prepared-plan blob to the permanent CUDA slab.

prepare_mega_pass_kernels

Compile/load every kernel that a prepared mega-pass may dispatch.

q4k_device_weight_count

How many Q4 SoA matrices are sticky-resident on CUDA.

q4k_weight_resident

q8_0_gemv_oracle_into

Decode a row-major GGML Q8_0 matrix and multiply it by x.

try_cuda_batch_gemv

Batch dense GEMM: for each of batch rows of hidden (n_in), compute out (n_out).

try_cuda_batch_gemv_cached

Like try_cuda_batch_gemv but uses a cached weight by fingerprint if present; otherwise caches weight under key then multiplies.

try_cuda_batch_gemv_cached_only

Lookup-only: run GEMV if key is already densified in the cache.

try_cuda_mega_pass

Run from a caller-provided f32 embedding. Kept as the portable compatibility entry point.

try_q4k_soa_attention_device

P4 device attention: sticky-x QKV (no intermediate D2H) → RoPE → device KV write → GQA SDPA → O-proj GEMV → one residual-delta readback.

try_q4k_soa_ffn_block

Full pre-norm FFN expansion+down on CUDA with one host→device x upload and one device→host readback (mid stays on-device between kernels).

try_q4k_soa_ffn_block_residual

Like try_q4k_soa_ffn_block but fuses residual on device: out = residual + FFN(x) with a single host readback.

try_q4k_soa_fused_swiglu

Fused SwiGLU expansion on two sticky Q4_K SoA weights (T-A2):
out[i] = silu(gate_row_i · x) * (up_row_i · x). One kernel, one x upload.

try_q4k_soa_gemv

On-device Q4_K SoA dequant-GEMV (type 112 / .soa.p64).

try_q4k_soa_qkv

Project Q, K, and V from the same activation with one sticky x upload and one fused QKV kernel (shared act tile; GQA: K/V only for row < n_kv).

try_q8_0_cuda_gemv

Execute one native CUDA Q8_0 GEMV. This public runner is a differential/profiling boundary; prepared whole-model decode uses the same kernel with pre-resident views.

warm_cuda_context

Ensure the multi-weight CUDA context exists (NVIDIA clocks / driver warm). Safe to call from portable paths — brings A2000 out of idle so wgpu resident decode sees production clocks (measured ~4× vs cold portable on 3B).

weight_cache_len

How many weight matrices are currently cached.

weight_fingerprint

FNV-1a over weight bytes + dims for cache key.