Expand description
Mixture-of-Experts (MoE) execution and offload engine (Work Packages F9, F10).
Provides zero-heap top-k routing, NVFP4 unpacking and GEMV, and heterogeneous GPU/CPU expert offload for the Qwen3.6-35B-A3B architecture on RTX A2000 12GB.
Re-exports§
pub use dispatch::dispatch_moe_step;pub use expert_cache::query_dynamic_accelerator_budget;pub use expert_cache::MoeOffloadManager;pub use ftw_loader::FtwExpertBankSlice;pub use ftw_loader::FtwManifest;pub use ftw_loader::FtwModelPackage;pub use ftw_loader::FtwShardEntry;pub use ftw_loader::FtwTensorEntry;
Modules§
- dispatch
- Compatibility adapter for the canonical Etalon routed-expert dispatcher.
- expert_
cache - Host memory observations over canonical expert-cache mechanics.
- ftw_
loader - FTW multi-shard loader is file/
memmap2-backed — native hosts only. FTW (FreeToken Weight) format loader and multi-shard memory mapper (Work Package F9/F10). - nvfp4
- Compatibility exports for the canonical Etalon numerical implementation.
- placement
- Canonical measured expert placement policy.
- routing
- Compatibility exports for the canonical Etalon numerical implementation.
Structs§
- Dynamic
Vram Status - Telemetry and allocation status for dynamic VRAM tiering.
- Expert
Cache Telemetry - Telemetry counters for MoE offload and residency.
- Expert
Placement Calibration - Measured timings used to choose CPU evaluation or a GPU staging transfer.
- Expert
Placement Policy - Frozen placement policy used for all expert-cache misses in a generation run.
- Expert
Residency Profile - Cold-path memory observation bound to a model activation.
- Expert
Weight View - Description of one expert’s raw NVFP4 matrices.
- FtwExpert
Data - Zero-copy byte slices for one expert’s NVFP4 tensors.
- MoeRouting
Config - Configuration for MoE routing.
Enums§
- MoeError
- Error conditions during MoE routing.
- Nvfp4
Error - Error conditions during NVFP4 unpacking.
- Personal
Hardware Tier - Device capability tier for personal computing local AI across Windows, Linux, and macOS.
- Slot
Access Outcome - Outcome of accessing an expert in the offload cache.
Constants§
- DEFAULT_
GPU_ EXPERT_ SLOTS - Maximum number of active expert slots in GPU VRAM simultaneously.
- E2M1_
TABLE - Standard E2M1 FP4 lookup table (16 entries: 1 sign, 2 exp, 1 mantissa).
- MAX_
INTERMEDIATE_ DIM - MAX_
MOE_ TOPK - Maximum number of routed experts per token supported on the stack.
- MAX_
ROUTED_ EXPERTS - NVFP4_
BLOCK_ SIZE - NVFP4 block size: each scale factor governs 16 weights (8 packed bytes).
- NVFP4_
BYTES_ PER_ BLOCK
Functions§
- combine_
expert_ outputs - Combine expert outputs using computed routing weights.
- compute_
principled_ slot_ capacity - Compute principled slot capacity given detected system memory and model footprint.
- dequantize_
nvfp4_ block - Dequantize a single 16-element block of NVFP4 weights.
- dequantize_
nvfp4_ row - Dequantize a row of packed NVFP4 weights into caller-supplied
out_row. - evaluate_
swiglu_ expert_ nvfp4 - Zero-heap evaluation of one SwiGLU expert: y = (SiLU(x * W_gate) * (x * W_up)) * W_down
- fp8_
e4m3_ to_ f32 - Convert an FP8 E4M3 byte to f32.
- nvfp4_
gemv_ zero_ heap - Compute Matrix-Vector multiplication $y = A \cdot x$ where $A$ is in NVFP4 format.
- query_
dynamic_ hardware_ tier - Query the current dynamic hardware tier given accelerator budget and workload requirements.
- route_
topk - Zero-heap top-k expert selection and routing weight calculation.
- silu
- SiLU (Swish) activation function: x * sigmoid(x).