Skip to main content

Module ragged_backend

Module ragged_backend 

Source
Expand description

Multi-sequence ragged decode backend with CUDA graph replay bucketing (Work Package F6).

Replaces test mocks with a zero-heap continuous-batching backend that supports:

  • Padded batch bucketing (1, 2, 4, 8) for fixed-shape CUDA graph capture/replay.
  • Strict isolation between real requests and dummy/padded rows.
  • Independent per-sequence KV block-table indexing and execution.
  • Deterministic execution receipts and device-to-host bandwidth accounting.

Structs§

MultiSequenceRaggedBackend
A zero-heap multi-sequence decode backend with padded graph bucketing.
RaggedBackendStats
Statistics and accounting for the ragged decode backend.

Constants§

GRAPH_BUCKET_SIZES
Standard power-of-two graph replay buckets.
MAX_BATCH_BUCKET
Maximum supported graph replay bucket size.
PADDED_REQUEST_ID
Sentinel request ID for padded dummy rows.
PADDED_SLOT_ID
Sentinel slot ID for padded dummy rows.

Functions§

select_graph_bucket
Find the smallest graph bucket size that fits batch_size.