Expand description
Client-facing, caller-buffered boundary for one native continuous-decode round.
This module intentionally does not own a model or KV cache. The Studio/client supplies stable session rows and caller-owned scratch; the native scheduler backend performs the actual token step. Keeping this boundary flat means a future chat state machine can use the same path without serialising per-turn requests through a global decode mutex.
Structs§
- Chat
Batch Session State - State for an admitted chat session advancing through continuous decode rounds.
- Chat
Continuous Batch Coordinator - Zero-heap-hot-path coordinator managing concurrent chat sessions across continuous batch decode rounds.
- Continuous
Batch Receipt - Receipt returned to the client/Studio telemetry path after a decode round.
- Session
Decode Input - One client session’s input to a native continuous decode round.
- Session
Decode Output - Token returned for one client session, preserving its scheduler identity.
Enums§
Constants§
- MAX_
CONCURRENT_ CHAT_ SESSIONS - Maximum concurrent active chat sessions in a single continuous batch round.
Functions§
- execute_
continuous_ batch_ round - Execute one real ragged decode round from client session rows.