Expand description
Streaming import path for RDF text sources → canonical .q42 volume.
Pipeline: a Rio parser on the main thread streams triples into a bounded channel; a pool of worker
shards hashes each triple into an NQuin and (in IngestMode::Complete) forwards first-seen
terms; a collector interns those terms into a capped spilling lexicon and writes quin runs; merge
publishes a volume set via crate::q42_volume::UnifiedVolumeBuilder — the GOVERNING .q42
layout (160-byte SuperBlock headers, block directory, BIDX, Merkle-DAG, and Q42LEX shards).
History: this path previously wrote headerless LZ4 blocks and an empty lexicon, so
Q42Volume::read_all_quins could not read the graph back and all literal text was discarded while
the shrunk file was reported as “compression”. Both defects are fixed — see IngestMode.
Re-exports§
pub use crate::query::ingest_job;pub use crate::query::ingest_report::format_bytes;pub use crate::query::ingest_report::format_count;pub use crate::query::ingest_report::IngestPhase;pub use crate::query::ingest_report::IngestReport;pub use crate::query::ingest_report::IngestSnapshot;pub use crate::query::ingest_formats::RawTriple;
Enums§
- Ingest
Mode - How much of the source graph the
.q42retains.
Functions§
- catalog_
base_ iri - Base IRI for relative terms / empty
xml:basein catalog RDF. - expand_
empty_ turtle_ prefixed_ names - Expand Turtle
prefix:empty local names (dc:,foaf:,ns1:) to IRIs. - streaming_
import_ rdf - Back-compat entry point: ingests losslessly (
IngestMode::Complete) — the honest default. - streaming_
import_ rdf_ volume_ set_ with_ mode - Stream-ingest RDF into a front-embedded logical volume root and immutable, size-capped child Q42 segments.
- streaming_
import_ rdf_ with_ job - Resume or run a durable job directory (
job.json+runs/). - streaming_
import_ rdf_ with_ mode - Stream-ingest an RDF source into a
.q42volume under an explicitIngestMode. - streaming_
import_ rdf_ with_ report - Same as the volume-set / single-file import, with a progress sink for CLI / UI.
- streaming_
import_ rdf_ with_ report_ filtered - Same as
streaming_import_rdf_with_report, with omit-predicate hashes applied before hashing (must matchverify-graph --omit-predicates). - verify_
integrity - Verifies
integrity.