pub fn serialize_string_lexicon(
entries: &HashMap<u64, String>,
) -> Result<Vec<u8>, LexError>Expand description
Serialize a hash → string map into the canonical Q42LEX byte layout that Q42LexMmap reads
back (magic, sorted 16-byte index, tagged string blob). This is the write side of the lexicon:
the RDF ingest calls it in lossless (“Complete”) mode so every subject/predicate URI and every
literal is recoverable via Q42LexMmap::lookup_hash, instead of being hashed away.
Strings are stored as UTF-8 (LEX_TAG_STRING), preserving the full Unicode range — this is
essential, not cosmetic: a lexicon that only kept ASCII would silently erase most of the world’s
languages (WordNet alone carries Finnish, Thai, and dozens more). A string longer than the 16-bit
length field can express is truncated at a UTF-8 character boundary (never mid-codepoint), so a
multi-byte grapheme is never split into invalid bytes.
The map is keyed by the same value stored in the quin field (for objects that is the
OBJECT_HASH_MASK-masked hash), so lookup_hash(quin.object) resolves directly. Entries are sorted
by hash for the reader’s binary search; the caller’s map has already de-duplicated by hash.
Returns LexError::TermTooLong rather than silently truncating a term.