Module q42_volume

Expand description

Unified .q42 v3 volume — lexicon, block index, and LZ4-compressed SuperBlocks in one file.

Layout (all little-endian):

[0..256)              Q42VolumeHeader
[lex_offset ..]       Q42LEX blob (uncompressed)
[bidx_offset ..]      BIDX blob (uncompressed)
[block_dir_offset ..] block_count × BlockDirectoryEntry (16 bytes each)
[data_offset ..]      concatenated LZ4 block payloads (lz4_flex prepend_size)

Legacy v1 sidecars (.q42.lex, .q42.bidx) and separate .c.q42 transport files are deprecated; new ingest writes v3 only.

Structs§

BidxBlockRange: A contiguous half-open block interval selected by an object hash.
BidxMatchPage: One caller-buffered page from a BIDX match interval.
BlockDirectoryEntry
BlockFieldPostings: One SuperBlock’s compact S/P/C membership.
CidSha256: A verified content identifier. Only sha2-256 (32-byte) digests are accepted.
ClassificationCounts
HttpRangeSource: Native HTTP byte-range source for trusted gateways. It never accepts a 200 OK full-body fallback: large Q42 access must stay range-bounded.
IpfsGatewaySegmentFactory: Opens manifest ipfs://CID child locators through one configured gateway. The CID and byte length remain manifest-committed; the gateway is merely a transport route and cannot substitute a different-length object.
LocalFileRangeSource: Native local-file implementation. The mutex serialises seek/read pairs; high-concurrency remote adapters should instead use independent requests.
OpfsCallbackRangeSource: Host fill callback: (offset, length, buf) -> bytes_written.
OpfsSliceRangeSource: Caller-buffered view of host-supplied bytes (slice or owned entity).
Q42BlockCursor: Sequential, caller-buffered view of a Q42 volume’s SuperBlocks.
Q42BlockMeta: Metadata for one decoded SuperBlock returned by Q42BlockCursor.
Q42ByteRange: A half-open byte interval, represented without an end-offset overflow.
Q42InspectReport: Machine-readable inspect receipt. Empty lexicon and missing postings are named, not papered over.
Q42LexiconSegment: Immutable physical Q42 file holding a contiguous hash range of the shared Q42LEX dictionary. It has no graph blocks: terms remain front-embedded in a Q42 container and can be retrieved by exact ranges independently of data segment size.
Q42Magnet: A single-file magnet for one immutable Q42 artifact.
Q42ObjectMatchPage: One page of exact object matches written by Q42RangeVolume::find_object_into.
Q42ObjectSearchCursor: Resume state for a caller-buffered object search. It is tied to the object hash passed to Q42RangeVolume::find_object_into.
Q42PublicationVerdict
Q42RangeQueryCursor: Resume state for a bounded range-query page.
Q42RangeQueryPage
Q42RangeQueryPattern: A simple physical pattern for range-backed Q42 scans. None is an unbound SPARQL position. The planner selects the BIDX object index when object is bound and otherwise performs a bounded sequential block scan.
Q42RangeQueryPlan
Q42RangeVolume: A Q42 reader that fetches exactly the bytes needed from a random-access source. All variable-size buffers remain caller-owned.
Q42RangeVolumeSet: Transport-neutral logical Q42 snapshot. Root and child volumes can be opened from HTTP/IPFS range sources just as from a local file.
Q42RolloverPublisher: Writes object-sorted SuperBlocks into successive .q42 children and publishes an atomic root catalog when finished.
Q42SectionReport: One named byte interval inside the file.
Q42SegmentMatchPage: One caller-buffered page from a manifest object-hash segment match.
Q42SegmentMatchRange: A contiguous interval of volume segments that can contain one object hash.
Q42VerificationReceipt: Exhaustive cold-path verification receipt for one physical Q42 segment.
Q42VerifyReceipt
Q42VerifySetReport: One root plus every physical child named by its catalog.
Q42Volume: Memory-mapped unified v2 volume reader.
Q42VolumeHeader: Q42 volume header — 256 bytes, repr(C, packed). v3 builds hard-reject files with version < 3 — run q42 migrate meta first.
Q42VolumeManifest: Front-embedded catalog which makes immutable Q42 segments one snapshot.
Q42VolumeSegment: One immutable child segment in a logical Q42 volume.
Q42VolumeSet: Locally backed multi-segment query snapshot. It deliberately has no remote transport yet: IPFS/HTTP range sources will implement the same contract.
Q42VolumeSetMagnets: Root + children, each with its own magnet (one info-hash per file).
Q42VolumeSetQueryCursor: Resume state for a range-backed scan across a logical Q42 volume set.
Q42VolumeSetQueryPage
StreamingQ42VolumeWriter: Writes a sorted Q42 segment without retaining its compressed payload or per-block metadata in heap collections. Temporary streams are RAII-owned.
StreamingVolumeAppender: Streaming append-only interface for Q42 Unified Volumes. Allows continuous block accumulation without loading the entire volume in memory.
UnifiedVolumeBuilder: Incremental builder for large external-sort merges (one SuperBlock at a time).
VerifiedCarBlock: One verified raw-leaf block from a CARv1 stream.
VerifyCheck

Enums§

CheckStatus
PublicationIntent: What the caller asserts about a file. Never overrides Quin-level rights.
Q42PublicationClass: How this volume may move, if at all.
Q42QueryMode
Q42RangeQueryStrategy
Q42Transport: Where a classified volume is allowed to travel.
VerifyLevel

Constants§

BIDX_MAGIC
DEFAULT_SEGMENT_MAX_BYTES: Default child-segment cap (512 MiB). Includes header, lexicon, indexes, data.
FIELD_POSTINGS_MAGIC
FIELD_RANGE_INDEX_ENTRY_BYTES
FIELD_RANGE_INDEX_HEADER_BYTES
FIELD_RANGE_INDEX_MAGIC
FLAG_BLOCKS_LZ4
FLAG_FIELD_POSTINGS: Compact per-block S/P/C postings (or measured Bloom) are present.
FLAG_FIELD_RANGES: A per-SuperBlock subject/predicate/context range index is present in front matter. It supplements (but does not replace) the object-sorted BIDX.
FLAG_OBJECT_SORTED
FLAG_PERMISSIVE_COMMONS: Affirmative Permissive Commons catalog. Required (with a clean Quin scan) before a public magnet / HTTP web-seed / IPFS pin is emitted.
FLAG_SANCTUARY: Affirmative Sanctuary / Selfhood volume. Public hash addressing is denied. Writers also set this when any Quin is restricted, classified, medical, legal, fiduciary, or bilateral.
FLAG_VOLUME_ROOT
HEADER_SIZE
MAX_COMPRESSED_SUPERBLOCK_SIZE: Conservative caller-buffer bound for an LZ4 prepend_size encoding of one Q42 SuperBlock. The bound includes its four-byte decoded-size prefix.
MAX_VOLUME_MANIFEST_BYTES
Q42_MAGIC
Q42_VERSION_V3
QUIN_SIZE
RESIDENT_QUERY_MAX_BYTES: 4 MiB decoded-file threshold. Above this, callers must not read_all_quins.
SUPERBLOCK_HEADER
SUPERBLOCK_SIZE

Traits§

Q42LexiconRangeFactory: Opens a manifest-attested Q42LEX shard through a range transport.
Q42RangeSource: Exact, caller-buffered random access. Implementations must fail if the source returns a short response; partial data is never treated as success.
Q42SegmentRangeFactory: Opens an immutable child source identified by the front-embedded manifest. Construction may allocate; all subsequent block reads remain caller-buffered.

Functions§

append_segment_to_root: Append a finished child segment to an existing root by publishing a new generation (stem-root.q42 overwritten atomically via temp + rename).
bidx_blocks_for_hash: BIDX binary search (shared with sidecar format).
classify_q42_path
classify_q42_volume
classify_q42_volume_set: Volume-set: any child that cannot be public denies the whole public set.
compact_volume_set: Compact root into out_dir.
compose_magnet
decode_and_verify_car: Decode a CARv1 buffer, verify every block CID, and concatenate raw (0x55) leaves as the UnixFS entity. Non-raw blocks are verified but not appended (dag-pb roots are proofs, not Q42 bytes).
decode_superblock_quins: Live Quins in a decompressed SuperBlock (160-byte header + 48-byte slots).
deny_public_publication
encode_bidx: Encode BIDX bytes from per-block min/max object hashes.
encode_block_postings: Encode one SuperBlock’s three fields. Chooses Bloom when it is strictly smaller than the exact posting list.
encode_field_range_index: Encode conservative per-block ranges for the non-object triple fields. Every range encloses all values in its block, so a range miss is safe to skip while a hit still requires normal Quin-level matching.
encode_lex: Encode Q42LEX bytes from a hash → string map.
encode_lex_with_entries: Encode Q42LEX bytes from a hash → LexiconEntry map (supports embedded triples).
encode_postings_section
encode_raw_car: Encode a one-block raw CARv1 used by tests and fixtures.
encode_superblock: Encode Q42LEX bytes from a hash → LexiconEntry map (supports embedded triples).
extract_entity_bytes: Extract a half-open entity-byte interval from verified raw leaves.
header_to_bytes
inclusive_entity_bytes: Inclusive entity-bytes=start:end (IPIP-0402) → half-open Q42 range.
ipfs_gateway_range_source
ipns_gateway_range_source: Construct a range source for an IPNS-published root. Resolution is bounded by the HTTP adapter’s 10-second connect and 30-second request timeouts; after the root manifest is read, child segments must use immutable CIDs.
is_unified_volume: Returns true if path begins with a unified volume header.
measure_bloom_false_positives: Measured false-positive rate of the Bloom encoding on a synthetic field.
migrate_v2_to_v3: Upgrade a v1/v2 .q42 file to v3 by rewriting the version field and zeroing the new extension fields in the 256-byte header. Data blocks are untouched.
quin_requires_sanctuary: True when this Quin is Selfhood, bilateral, or a protected ODRL tier.
root_relative_path
sha1_hex_file
validate_exact_range_response: Validate an HTTP/IPFS adapter response before it is accepted by a reader. content_range_start is the start declared by Content-Range or the verified CAR block offset; returned_len is its actual payload size.
verify_car_bytes_as_q42_source: Decode and CID-verify a CARv1 buffer, then expose raw-leaf entity bytes.
verify_local_car_as_q42_source: Read a local CARv1 path, CID-verify every block, expose entity bytes.
verify_source_sha256: Stream the complete immutable source through a caller-provided scratch buffer and compare it to the SHA-256 committed by the root descriptor. This is the bounded verification path for a trusted HTTP/IPFS gateway; partial-CAR block proofs can later authenticate individual ranges sooner.
verify_volume_set_from_root: Verify a standalone file, or a volume-set root and every named child.
write_sorted_quins_volume: Sort quins by object hash, chunk into SuperBlocks, and write a unified v3 volume.
write_sorted_quins_volume_with_author: Same as write_sorted_quins_volume, with a DAG author DID on each SuperBlock commit.
write_unified_volume: Write a unified v3 .q42 volume.
write_unified_volume_with_entries: Write a unified v3 .q42 volume with embedded triple support.
write_volume_root: Publish a root Q42 segment whose front matter catalogs immutable child segments. The root is intentionally data-empty; query code opens the catalog snapshot through Q42VolumeSet.
write_volume_root_for_commons: Same as write_volume_root, but marks the catalog as Permissive Commons. Use for public ontologies / knowledge graphs, never for personal volumes.
write_volume_root_with_lex: Publish a logical-volume root with a lossless shared Q42LEX in its front matter. Child data segments may keep empty local lexicons because all term resolution for the snapshot is recoverable from this immutable root.

Type Aliases§

VerifiedCarRangeSource: Verified UnixFS entity reconstructed from a local CARv1 file.

Module q42_volume

Module q42_volume Copy item path

Structs§

Enums§

Constants§

Traits§

Functions§

Type Aliases§

Module q42_volume