Skip to main content

Module expert_cache

Module expert_cache 

Source
Expand description

Host memory observations over canonical expert-cache mechanics.

Structs§

DynamicVramStatus
Telemetry and allocation status for dynamic VRAM tiering.
ExpertCacheTelemetry
Telemetry counters for MoE offload and residency.
ExpertResidencyProfile
Cold-path memory observation bound to a model activation.
MoeOffloadManager

Enums§

PersonalHardwareTier
Device capability tier for personal computing local AI across Windows, Linux, and macOS.
SlotAccessOutcome
Outcome of accessing an expert in the offload cache.

Constants§

DEFAULT_GPU_EXPERT_SLOTS
Maximum number of active expert slots in GPU VRAM simultaneously.

Functions§

compute_principled_slot_capacity
Compute principled slot capacity given detected system memory and model footprint.
query_dynamic_accelerator_budget
Query dynamic accelerator budget and memory architecture (discrete vs unified).
query_dynamic_hardware_tier
Query the current dynamic hardware tier given accelerator budget and workload requirements.