Expand description
Durable ingest jobs: inspect, compare, resume, append, and attested streams.
A job directory survives process death (unlike tempfile::TempDir). Incomplete
OSM-style runs can be reviewed and, when the source is seekable and uncompressed,
continued. HTTP/gzip streams are ingested without a second copy of the source;
SHA-256 of the decompressed bytes plus 16 MiB window hashes remain so the
original can be verified later without keeping the file.
Structs§
- Compare
Report - Digest
Outcome - Digesting
Reader - Ingest
Inspect Report - Ingest
Job - Ingest
JobSpec - Ingest
JobStatus - Opened
Source - Source
Attestation
Enums§
Constants§
Functions§
- adopt_
legacy_ scratch - Copy a live/legacy scratch tree into a new job directory so it can be inspected and, if the source file is still there and uncompressed, continued.
- append_
rdf_ to_ root - Ingest
extrainto a sibling volume set, then graft its data + lexicon segments ontorootas a new generation. The extra source is streamed (URL/gzip allowed); it is not kept. - compare_
attestation_ file_ to_ path - Compares an attestation file to a path.
- compare_
attestation_ to_ stream - Compares attestation to a stream.
- continue_
job - Continue is implemented by
streaming_import_rdf_with_job(adopt + skip-N). - detect_
encoding - Detects ingest encoding from filename,
Content-Encoding, and magic bytes. - hex_
encode - Lowercase hex encoding (no external deps).
- infer_
rdf_ format - Infers RDF format from the filename extension.
- inspect_
job - Inspects an ingest job.
- inspect_
legacy_ scratch - Review a TempDir-style scratch (the live OSM job): chunk_.tmp + lexrun_.tmp.
- inspect_
volume_ root - Inspects a volume root.
- job_
status ?::job_status.- open_
ingest_ source - Opens and wraps the source (decompression + digesting reader) per kind/encoding.
- publish_
job - Publish already-hashed quin/lex runs to the job’s
.q42root. Does not re-read the Turtle (the OSM continue died on disk during this step). - read_
progress_ json - Reads
progress-json. - read_
window_ hashes - Reads all window-hash lines, hex-decoding each; empty vec when absent.
- resume_
is_ supported - A job can continue when it is not complete and we still have the source locator. Turtle/HTTP resume re-reads from the start and skips already accepted triples (correct; prefixes stay valid). Gzip/zstd HTTP costs another download. There is no mid-stream inflater checkpoint in v1.
- unix_
now - Current UNIX epoch seconds (saturating at 0 on clock skew).
- window_
commitment - SHA-256 over
WINDOW_BYTES+ window count + each hash — binds the commitment to the segmentation scheme. - write_
json_ atomic - Serializes
valueto<path>.json.tmpthen renames — readers never see a partial file.