Skip to main content

Module ingest

Module ingest 

Source
Expand description

Streaming import path for RDF text sources → canonical .q42 volume.

Pipeline: a Rio parser on the main thread streams triples into a bounded channel; a pool of worker shards hashes each triple into an NQuin and (in IngestMode::Complete) forwards first-seen terms; a collector interns those terms into a capped spilling lexicon and writes quin runs; merge publishes a volume set via crate::q42_volume::UnifiedVolumeBuilder — the GOVERNING .q42 layout (160-byte SuperBlock headers, block directory, BIDX, Merkle-DAG, and Q42LEX shards).

History: this path previously wrote headerless LZ4 blocks and an empty lexicon, so Q42Volume::read_all_quins could not read the graph back and all literal text was discarded while the shrunk file was reported as “compression”. Both defects are fixed — see IngestMode.

Re-exports§

pub use crate::query::ingest_job;
pub use crate::query::ingest_report::format_bytes;
pub use crate::query::ingest_report::format_count;
pub use crate::query::ingest_report::IngestPhase;
pub use crate::query::ingest_report::IngestReport;
pub use crate::query::ingest_report::IngestSnapshot;
pub use crate::query::ingest_formats::RawTriple;

Enums§

IngestMode
How much of the source graph the .q42 retains.

Functions§

catalog_base_iri
Base IRI for relative terms / empty xml:base in catalog RDF.
expand_empty_turtle_prefixed_names
Expand Turtle prefix: empty local names (dc:, foaf:, ns1:) to IRIs.
streaming_import_rdf
Back-compat entry point: ingests losslessly (IngestMode::Complete) — the honest default.
streaming_import_rdf_volume_set_with_mode
Stream-ingest RDF into a front-embedded logical volume root and immutable, size-capped child Q42 segments.
streaming_import_rdf_with_job
Resume or run a durable job directory (job.json + runs/).
streaming_import_rdf_with_mode
Stream-ingest an RDF source into a .q42 volume under an explicit IngestMode.
streaming_import_rdf_with_report
Same as the volume-set / single-file import, with a progress sink for CLI / UI.
streaming_import_rdf_with_report_filtered
Same as streaming_import_rdf_with_report, with omit-predicate hashes applied before hashing (must match verify-graph --omit-predicates).
verify_integrity
Verifies integrity.