Module hetero_dispatch 

Vendor-neutral heterogeneous compute dispatch + storage / precision policy.

This module replaces the former CUDA/cuFile GPUDirect-Storage bridge (cuda_bridge.rs, removed). That bridge was the engine’s one vendor-locked appendage — NVIDIA-only, Linux-only, and unverifiable without specific hardware. Removing it dissolves the hardware boundary entirely. The four capabilities it was meant to provide are delivered here on the engine’s GENERAL stack — portable wgpu (super::gpu) + memory-mapped I/O (super::host) — so they build, run, and verify on any backend (Vulkan / DX12 / Metal / WebGPU), no vendor SDK:

1. Unified-memory zero-copy (ZeroCopyStrategy) — choose mmap-direct on integrated / unified-memory GPUs (Apple Silicon via Metal, which wgpu maps transparently — CPU and GPU share physical RAM, so the mmap’d region is GPU visible with no copy) vs a one-time staging upload on discrete GPUs.
2. Hardware-agnostic fallback dispatcher (HeterogeneousDispatcher) — route a job to the GPU (wgpu) when it fits, else NPU, else CPU; and tile a matmul across passes when VRAM is exhausted instead of hard-failing.
3. Kernel stream fusion (plan_fusion) — group consecutive same-shape element-wise tensor ops into a single wgpu compute pass (the portable analogue of CUDA stream fusion; the engine’s fused shaders already do this at the shader level). Fewer passes ⇒ fewer dispatch / PCIe round-trips.
4. Mixed-precision policy (select_precision) — pick f32/f16/q8/q4 from the host’s VRAM / power / thermal budget.

The one capability deliberately NOT ported

NVIDIA GPUDirect-Storage’s true NVMe→VRAM DMA (bypassing system RAM) has no portable wgpu equivalent. The vendor-neutral substitute (and the engine’s actual path) is mmap + OS page cache + a one-time staging upload — zero-heap, standard OS mechanics, identical across an A2000 / Apple M-series / generic Linux box. GDS-class throughput only matters when streaming a 70B model off an NVMe array into an 80 GB datacenter GPU — the deployment the affordability rail explicitly does not target, so nothing on the critical path is lost.

Future, optional Vulkan zero-copy fast-path (documented, NOT built)

If a specific deployment ever justifies skipping the staging copy, the vendor-neutral way is a wgpu-hal Vulkan fast-path that imports the mmap’d weights as device memory via VK_EXT_external_memory_host (broadly supported, cross-vendor) + Resizable-BAR — lit up ONLY on the Vulkan backend, behind a vulkan_zero_copy_import feature, additive over the portable path. It is deliberately unbuilt: it needs unsafe wgpu-hal and is backend-specific (DX12/Metal have their own external-memory mechanisms), which would re-introduce exactly the per-backend coupling that removing CUDA just eliminated. Build it only when a real deployment needs it; never as the core storage dependency.

All routines here are pure-scalar policy / planning logic — zero heap, no recursion, run anywhere.

Structs

HeterogeneousDispatcher

Routes compute jobs across GPU / NPU / CPU and degrades gracefully under VRAM pressure instead of hard-failing (the behaviour the CUDA bridge lacked).

HostCapabilities

What the host can offer the dispatcher.

PowerThermalBudget

The host’s power / thermal / memory envelope for the precision decision.

TensorOp

One op in a planned dispatch sequence. shape is a shape token; element-wise ops only fuse with neighbours of the same shape.

Enums

ComputeBackend

The compute backend a job is routed to.

Precision

Numeric precision for weights / activations.

TensorOpKind

The fusability class of a tensor op in a dispatch sequence.

ZeroCopyStrategy

How weight data reaches the GPU for a given device.

Functions

plan_fusion

Plan stream fusion: the number of wgpu compute passes a sequence needs after fusing each maximal run of same-shape element-wise ops into a single pass. Barriers each take their own pass. Result is in 1..=ops.len(); fewer passes ⇒ fewer dispatch / PCIe round-trips. Zero-heap (single linear scan).

select_precision

Pick the precision tuned to the host’s thermodynamic / power / memory constraints: choose the coarsest precision that still fits param_count weights in the VRAM budget, and — when power or thermal headroom is tight — never run the heavy precisions (f32/f16 draw more power and generate more heat). Returns Q4 if even q4 overflows VRAM (the caller must then tile or offload).