Skip to main content

Module prefix_cache

Module prefix_cache 

Source
Expand description

Typed, bounded prefix KV cache for the QualiaDB local inference engine.

Replaces legacy unsafe 8-byte hash keying with an exact compatibility contract:

  • Model instance & tokenizer identity
  • Exact token sequence digest & token length
  • Full graph-context digest (not truncated to 8 bytes)
  • KV cache geometry & layer count
  • Strict completion flag (incomplete/cancelled prefill is never cached or restored)
  • Bounded capacity with deterministic LRU eviction conforming to the Sentinel ceiling

Structs§

BoundedInferenceCache
Bounded in-memory KV cache with capacity limits and deterministic LRU eviction.
InferenceCacheEntry
Cached KV data entry with metadata and completion invariant.
InferenceCacheKey
Strongly-typed cache key requiring exact compatibility across model, tokens, graph, and tensor geometry.

Constants§

DEFAULT_MAX_BYTES
Default maximum memory ceiling for cached KV floats (64 MiB).
DEFAULT_MAX_ENTRIES
Default maximum number of cached prompt KV sequences.

Functions§

digest_prompt_tokens
Compute a 256-bit collision-resistant digest over a slice of prompt token IDs.
get_prefix_cache
Access the process-wide bounded inference cache.