Expand description
Typed, bounded prefix KV cache for the QualiaDB local inference engine.
Replaces legacy unsafe 8-byte hash keying with an exact compatibility contract:
- Model instance & tokenizer identity
- Exact token sequence digest & token length
- Full graph-context digest (not truncated to 8 bytes)
- KV cache geometry & layer count
- Strict completion flag (incomplete/cancelled prefill is never cached or restored)
- Bounded capacity with deterministic LRU eviction conforming to the Sentinel ceiling
Structs§
- Bounded
Inference Cache - Bounded in-memory KV cache with capacity limits and deterministic LRU eviction.
- Inference
Cache Entry - Cached KV data entry with metadata and completion invariant.
- Inference
Cache Key - Strongly-typed cache key requiring exact compatibility across model, tokens, graph, and tensor geometry.
Constants§
- DEFAULT_
MAX_ BYTES - Default maximum memory ceiling for cached KV floats (64 MiB).
- DEFAULT_
MAX_ ENTRIES - Default maximum number of cached prompt KV sequences.
Functions§
- digest_
prompt_ tokens - Compute a 256-bit collision-resistant digest over a slice of prompt token IDs.
- get_
prefix_ cache - Access the process-wide bounded inference cache.