Expand description
Deterministic prefill and decode scheduling policies.
Provides deterministic fairness rules, bounded prefill chunk sizing, and prefill-vs-decode interleaving so long prefill operations cannot starve active decode requests.
Structsยง
- Prefill
Progress - Tracking structure for request prefill progress.
- Scheduling
Policy - Policy governing chunk limits and prefill-vs-decode interleaving.