Skip to main content

Module moe

Module moe 

Source
Expand description

Mixture-of-Experts (MoE) execution and offload engine (Work Packages F9, F10).

Provides zero-heap top-k routing, NVFP4 unpacking and GEMV, and heterogeneous GPU/CPU expert offload for the Qwen3.6-35B-A3B architecture on RTX A2000 12GB.

Re-exports§

pub use dispatch::dispatch_moe_step;
pub use expert_cache::query_dynamic_accelerator_budget;
pub use expert_cache::MoeOffloadManager;
pub use ftw_loader::FtwExpertBankSlice;
pub use ftw_loader::FtwManifest;
pub use ftw_loader::FtwModelPackage;
pub use ftw_loader::FtwShardEntry;
pub use ftw_loader::FtwTensorEntry;

Modules§

dispatch
Compatibility adapter for the canonical Etalon routed-expert dispatcher.
expert_cache
Host memory observations over canonical expert-cache mechanics.
ftw_loader
FTW multi-shard loader is file/memmap2-backed — native hosts only. FTW (FreeToken Weight) format loader and multi-shard memory mapper (Work Package F9/F10).
nvfp4
Compatibility exports for the canonical Etalon numerical implementation.
placement
Canonical measured expert placement policy.
routing
Compatibility exports for the canonical Etalon numerical implementation.

Structs§

DynamicVramStatus
Telemetry and allocation status for dynamic VRAM tiering.
ExpertCacheTelemetry
Telemetry counters for MoE offload and residency.
ExpertPlacementCalibration
Measured timings used to choose CPU evaluation or a GPU staging transfer.
ExpertPlacementPolicy
Frozen placement policy used for all expert-cache misses in a generation run.
ExpertResidencyProfile
Cold-path memory observation bound to a model activation.
ExpertWeightView
Description of one expert’s raw NVFP4 matrices.
FtwExpertData
Zero-copy byte slices for one expert’s NVFP4 tensors.
MoeRoutingConfig
Configuration for MoE routing.

Enums§

MoeError
Error conditions during MoE routing.
Nvfp4Error
Error conditions during NVFP4 unpacking.
PersonalHardwareTier
Device capability tier for personal computing local AI across Windows, Linux, and macOS.
SlotAccessOutcome
Outcome of accessing an expert in the offload cache.

Constants§

DEFAULT_GPU_EXPERT_SLOTS
Maximum number of active expert slots in GPU VRAM simultaneously.
E2M1_TABLE
Standard E2M1 FP4 lookup table (16 entries: 1 sign, 2 exp, 1 mantissa).
MAX_INTERMEDIATE_DIM
MAX_MOE_TOPK
Maximum number of routed experts per token supported on the stack.
MAX_ROUTED_EXPERTS
NVFP4_BLOCK_SIZE
NVFP4 block size: each scale factor governs 16 weights (8 packed bytes).
NVFP4_BYTES_PER_BLOCK

Functions§

combine_expert_outputs
Combine expert outputs using computed routing weights.
compute_principled_slot_capacity
Compute principled slot capacity given detected system memory and model footprint.
dequantize_nvfp4_block
Dequantize a single 16-element block of NVFP4 weights.
dequantize_nvfp4_row
Dequantize a row of packed NVFP4 weights into caller-supplied out_row.
evaluate_swiglu_expert_nvfp4
Zero-heap evaluation of one SwiGLU expert: y = (SiLU(x * W_gate) * (x * W_up)) * W_down
fp8_e4m3_to_f32
Convert an FP8 E4M3 byte to f32.
nvfp4_gemv_zero_heap
Compute Matrix-Vector multiplication $y = A \cdot x$ where $A$ is in NVFP4 format.
query_dynamic_hardware_tier
Query the current dynamic hardware tier given accelerator budget and workload requirements.
route_topk
Zero-heap top-k expert selection and routing weight calculation.
silu
SiLU (Swish) activation function: x * sigmoid(x).