Expand description
Multi-sequence ragged decode backend with CUDA graph replay bucketing (Work Package F6).
Replaces test mocks with a zero-heap continuous-batching backend that supports:
- Padded batch bucketing (1, 2, 4, 8) for fixed-shape CUDA graph capture/replay.
- Strict isolation between real requests and dummy/padded rows.
- Independent per-sequence KV block-table indexing and execution.
- Deterministic execution receipts and device-to-host bandwidth accounting.
Structs§
- Multi
Sequence Ragged Backend - A zero-heap multi-sequence decode backend with padded graph bucketing.
- Ragged
Backend Stats - Statistics and accounting for the ragged decode backend.
Constants§
- GRAPH_
BUCKET_ SIZES - Standard power-of-two graph replay buckets.
- MAX_
BATCH_ BUCKET - Maximum supported graph replay bucket size.
- PADDED_
REQUEST_ ID - Sentinel request ID for padded dummy rows.
- PADDED_
SLOT_ ID - Sentinel slot ID for padded dummy rows.
Functions§
- select_
graph_ bucket - Find the smallest graph bucket size that fits
batch_size.