Expand description
Host memory observations over canonical expert-cache mechanics.
Structs§
- Dynamic
Vram Status - Telemetry and allocation status for dynamic VRAM tiering.
- Expert
Cache Telemetry - Telemetry counters for MoE offload and residency.
- Expert
Residency Profile - Cold-path memory observation bound to a model activation.
- MoeOffload
Manager
Enums§
- Personal
Hardware Tier - Device capability tier for personal computing local AI across Windows, Linux, and macOS.
- Slot
Access Outcome - Outcome of accessing an expert in the offload cache.
Constants§
- DEFAULT_
GPU_ EXPERT_ SLOTS - Maximum number of active expert slots in GPU VRAM simultaneously.
Functions§
- compute_
principled_ slot_ capacity - Compute principled slot capacity given detected system memory and model footprint.
- query_
dynamic_ accelerator_ budget - Query dynamic accelerator budget and memory architecture (discrete vs unified).
- query_
dynamic_ hardware_ tier - Query the current dynamic hardware tier given accelerator budget and workload requirements.