Skip to main content

Module continuous_batch

Module continuous_batch 

Source
Expand description

Client-facing, caller-buffered boundary for one native continuous-decode round.

This module intentionally does not own a model or KV cache. The Studio/client supplies stable session rows and caller-owned scratch; the native scheduler backend performs the actual token step. Keeping this boundary flat means a future chat state machine can use the same path without serialising per-turn requests through a global decode mutex.

Structs§

ChatBatchSessionState
State for an admitted chat session advancing through continuous decode rounds.
ChatContinuousBatchCoordinator
Zero-heap-hot-path coordinator managing concurrent chat sessions across continuous batch decode rounds.
ContinuousBatchReceipt
Receipt returned to the client/Studio telemetry path after a decode round.
SessionDecodeInput
One client session’s input to a native continuous decode round.
SessionDecodeOutput
Token returned for one client session, preserving its scheduler identity.

Enums§

ContinuousBatchError

Constants§

MAX_CONCURRENT_CHAT_SESSIONS
Maximum concurrent active chat sessions in a single continuous batch round.

Functions§

execute_continuous_batch_round
Execute one real ragged decode round from client session rows.