Expand description
Bounded request queue and worker intake lifecycle.
Bridges asynchronous request intake to RequestScheduler and the sticky worker thread.
Guarantees:
- Bounded intake queue (rejects with
QueueFullrather than growing unbounded). - Exact lifecycle states:
Queued,Admitted,Prefill,Decode,Cancelled,Completed. - Cancellation at every phase has exactly one terminal result and clean currency release.
Structs§
- Bounded
Intake Queue - Bounded FIFO intake queue for the request scheduler.
- Intake
Request - Request descriptor within the bounded scheduler intake.
Enums§
- Intake
Error - Intake
State - Lifecycle state of a queued/running request.