Skip to main content

Module policy

Module policy 

Source
Expand description

Deterministic prefill and decode scheduling policies.

Provides deterministic fairness rules, bounded prefill chunk sizing, and prefill-vs-decode interleaving so long prefill operations cannot starve active decode requests.

Structsยง

PrefillProgress
Tracking structure for request prefill progress.
SchedulingPolicy
Policy governing chunk limits and prefill-vs-decode interleaving.