Skip to main content

Module q42_lex_ns

Module q42_lex_ns 

Source
Expand description

Q42LEX v4 writer: paged lexicon with per-page namespace dictionaries. Q42LEX v4 — paged lexicon with per-page namespace dictionaries.

Ownership note (AGENTS.md §0-B): the v3 codec is decomposed out of q42_lex.rs (which stays the reader state machine) so each file keeps a single purpose. The reader additions for v3 live in q42_lex.rs because they share the page-walk state; the writer and namespace economics live here. The version word is 4, aligned with the Q42 volume generation that first ships it — there is no LEX v3 (unreleased software; see ADR 0015).

§Why namespaces are stored once per page

RDF term strings are dominated by a small set of repeated namespace prefixes (https://schema.org/, http://www.w3.org/2000/01/rdf-schema#, did:q42:, urn:q42:, …). v2 pages store every term verbatim, so each term re-pays its namespace bytes. v4 keeps a namespace table inside each page and encodes qualifying terms as (namespace id, local name).

The table is page-local by design: a page is the unit an HTTP-Range / IPFS reader fetches, so every page stays independently decodable and the range-query contract of the v2 layout is preserved exactly. A volume may contain many ontologies, and an ontology may declare many namespaces — each page simply carries whichever namespaces its own terms use.

§Layout (v4 page, little-endian)

page header (16 B): [u32 entry_count][u32 ns_count][u64 blob_offset]
index:              entry_count × [u64 hash][u64 relative-from-blob-start]
blob:               ns table:  ns_count × [u16 len][utf-8 namespace]
                    entries:   0x01 [u16 len][utf-8]                    (verbatim)
                               0x04 [u16 ns_id][u16 local_len][utf-8]    (namespaced)

ns_count occupies the u32 slot v2 writers zeroed, so v2 pages read back with ns_count == 0 and the reader fails closed on any 0x04 entry in a v2 page. Old binaries that only know versions 1–2 reject v4 headers up front (BadIndex) instead of misparsing.

§Namespace extraction

namespace_split is scheme-agnostic: it splits at the final /, #, or : (all ASCII, so a UTF-8 codepoint is never split). The namespace keeps its trailing separator. This uniformly covers http(s) IRIs, did: methods (e.g. did:q42:, did:web:), urn:, and any future scheme whose terms end in a separator-delimited local name.

§When a term is encoded namespaced

Only when it strictly shrinks the page. Per namespace with m member terms of length n, the saving is m·(n − 2) − (n + 2) bytes (each namespaced entry pays a 2-byte ns_id instead of the namespace bytes, the namespace itself is paid once, both sides pay their u16 lengths). The rule is deterministic, so identical input maps produce identical bytes.

Functions§

namespace_split
Split a term into (namespace, local) at the final /, #, or :. The namespace includes its trailing separator; the local name is the rest. Returns None when the term has no separator — verbatim literals stay verbatim.
serialize_namespaced_paged_lexicon
Serialize a hash → string map into the namespaced paged Q42LEX v4 byte layout that crate::q42_lex::Q42LexMmap reads back.