Module thermal_telemetry 

W7 — real GPU thermal/power telemetry + a detect-and-recommend thermal governor.

orchestrator::ThermalGovernor previously had only a simulated implementation (CalculusThermalGovernor, a Newton-cooling ODE). This module adds a REAL one backed by NVIDIA NVML (via the optional nvml feature / nvml-wrapper), reading actual GPU temperature and power.

Policy: detect + recommend, never silently escalate. The governor maps live temperature to ThermalStatus and exposes a recommended TDP cap; it does NOT change the GPU’s power limit. Enforcement is a separate, explicit, privileged opt-in (NvmlThermalGovernor::apply_power_limit_w) that nothing here calls automatically — a human/admin policy must invoke it. This is the in-repo form of the human-centric-control norm for the off-grid / constrained-power target: the machine reports and recommends; the human decides.

When the nvml feature is off, or NVML/the driver is absent (non-NVIDIA host), telemetry degrades cleanly: sample_gpu_thermal() returns None and open_thermal_governor() returns the NullThermalGovernor (always Cool), so callers never need to know whether NVML is present.

Structs

GpuThermalSample

A point-in-time GPU thermal/power reading.

Enums

ThermalAction

What the governor should do this check — the pure decision, separated from the NVML side effects so it is unit-testable. critical_streak counts consecutive Critical checks INCLUDING this one; capped is whether we currently hold an applied cap.

Constants

CRITICAL_DWELL

Number of consecutive Critical checks before auto-cap engages (hysteresis — ignores transient spikes). At the decode-loop tick cadence (~every 64 tokens) this is several seconds of sustained Critical.

CRITICAL_THRESHOLD_C

WARM_THRESHOLD_C

GPU temperature bands (°C) — the same thresholds the simulated CalculusThermalGovernor uses, so the real and simulated governors classify identically.

Functions

decide_thermal_action

Pure enforcement decision (see ThermalAction). Hysteresis: only caps on sustained Critical, holds the cap through Warm, and restores only once back to Cool.

gpu_auto_cap_enabled

Whether automatic TDP capping is enabled (env var wins if set, else the runtime flag). Read this to reflect the current mode in a UI.

open_thermal_governor

Construct the best available thermal governor: the real NVML one when the nvml feature is on and NVML initializes, else the NullThermalGovernor (always Cool). Mirrors open_storage / open_platform_filter — callers don’t branch on platform/feature.

sample_gpu_thermal

Read one GPU thermal/power sample. None when the nvml feature is off or NVML/the driver is unavailable (non-NVIDIA host, no driver). Never panics. Suitable for a UI telemetry poll.

set_gpu_auto_cap

Enable/disable automatic TDP capping under sustained Critical temperature (QUALIA_LLM_GPU_AUTO_CAP). Runtime mode switch — the desktop UI / host calls this. Default ON (effective only when an NVIDIA card + NVML are present).

status_for_temp

Map a GPU temperature to the project ThermalStatus.

thermal_tick

Periodic thermal check for the decode hot path — samples a RESIDENT NVML governor and, per the auto-cap user option, enforces (sustained Critical → apply the recommended cap; cooled → restore) or recommends. Cheap (an NVML read); call every N decode tokens. No-op when the nvml feature is off or NVML/the driver is unavailable. The resident governor keeps the auto-cap hysteresis state (Critical streak, capped flag, saved limit) across ticks.