Expand description
Q42LEX v4 writer: paged lexicon with per-page namespace dictionaries. Q42LEX v4 — paged lexicon with per-page namespace dictionaries.
Ownership note (AGENTS.md §0-B): the v3 codec is decomposed out of
q42_lex.rs (which stays the reader state machine) so each file keeps a
single purpose. The reader additions for v3 live in q42_lex.rs because
they share the page-walk state; the writer and namespace economics live
here. The version word is 4, aligned with the Q42 volume generation that
first ships it — there is no LEX v3 (unreleased software; see ADR 0015).
§Why namespaces are stored once per page
RDF term strings are dominated by a small set of repeated namespace
prefixes (https://schema.org/, http://www.w3.org/2000/01/rdf-schema#,
did:q42:, urn:q42:, …). v2 pages store every term verbatim, so each
term re-pays its namespace bytes. v4 keeps a namespace table inside each
page and encodes qualifying terms as (namespace id, local name).
The table is page-local by design: a page is the unit an HTTP-Range / IPFS reader fetches, so every page stays independently decodable and the range-query contract of the v2 layout is preserved exactly. A volume may contain many ontologies, and an ontology may declare many namespaces — each page simply carries whichever namespaces its own terms use.
§Layout (v4 page, little-endian)
page header (16 B): [u32 entry_count][u32 ns_count][u64 blob_offset]
index: entry_count × [u64 hash][u64 relative-from-blob-start]
blob: ns table: ns_count × [u16 len][utf-8 namespace]
entries: 0x01 [u16 len][utf-8] (verbatim)
0x04 [u16 ns_id][u16 local_len][utf-8] (namespaced)ns_count occupies the u32 slot v2 writers zeroed, so v2 pages read back
with ns_count == 0 and the reader fails closed on any 0x04 entry in a
v2 page. Old binaries that only know versions 1–2 reject v4 headers up
front (BadIndex) instead of misparsing.
§Namespace extraction
namespace_split is scheme-agnostic: it splits at the final /, #, or
: (all ASCII, so a UTF-8 codepoint is never split). The namespace keeps
its trailing separator. This uniformly covers http(s) IRIs, did:
methods (e.g. did:q42:, did:web:), urn:, and any future scheme whose
terms end in a separator-delimited local name.
§When a term is encoded namespaced
Only when it strictly shrinks the page. Per namespace with m member
terms of length n, the saving is m·(n − 2) − (n + 2) bytes (each
namespaced entry pays a 2-byte ns_id instead of the namespace bytes, the
namespace itself is paid once, both sides pay their u16 lengths). The
rule is deterministic, so identical input maps produce identical bytes.
Functions§
- namespace_
split - Split a term into
(namespace, local)at the final/,#, or:. The namespace includes its trailing separator; the local name is the rest. ReturnsNonewhen the term has no separator — verbatim literals stay verbatim. - serialize_
namespaced_ paged_ lexicon - Serialize a hash → string map into the namespaced paged
Q42LEXv4 byte layout thatcrate::q42_lex::Q42LexMmapreads back.