Skip to main content

Module ingest_job

Module ingest_job 

Source
Expand description

Durable ingest jobs: inspect, compare, resume, append, and attested streams.

A job directory survives process death (unlike tempfile::TempDir). Incomplete OSM-style runs can be reviewed and, when the source is seekable and uncompressed, continued. HTTP/gzip streams are ingested without a second copy of the source; SHA-256 of the decompressed bytes plus 16 MiB window hashes remain so the original can be verified later without keeping the file.

Structs§

CompareReport
DigestOutcome
DigestingReader
IngestInspectReport
IngestJob
IngestJobSpec
IngestJobStatus
OpenedSource
SourceAttestation

Enums§

IngestEncoding
IngestJobPhase
IngestRdfFormat
IngestSourceKind

Constants§

JOB_CHECKPOINT
JOB_PROGRESS
JOB_SPEC
WINDOW_BYTES

Functions§

adopt_legacy_scratch
Copy a live/legacy scratch tree into a new job directory so it can be inspected and, if the source file is still there and uncompressed, continued.
append_rdf_to_root
Ingest extra into a sibling volume set, then graft its data + lexicon segments onto root as a new generation. The extra source is streamed (URL/gzip allowed); it is not kept.
compare_attestation_file_to_path
Compares an attestation file to a path.
compare_attestation_to_stream
Compares attestation to a stream.
continue_job
Continue is implemented by streaming_import_rdf_with_job (adopt + skip-N).
detect_encoding
Detects ingest encoding from filename, Content-Encoding, and magic bytes.
hex_encode
Lowercase hex encoding (no external deps).
infer_rdf_format
Infers RDF format from the filename extension.
inspect_job
Inspects an ingest job.
inspect_legacy_scratch
Review a TempDir-style scratch (the live OSM job): chunk_.tmp + lexrun_.tmp.
inspect_volume_root
Inspects a volume root.
job_status
?::job_status.
open_ingest_source
Opens and wraps the source (decompression + digesting reader) per kind/encoding.
publish_job
Publish already-hashed quin/lex runs to the job’s .q42 root. Does not re-read the Turtle (the OSM continue died on disk during this step).
read_progress_json
Reads progress-json.
read_window_hashes
Reads all window-hash lines, hex-decoding each; empty vec when absent.
resume_is_supported
A job can continue when it is not complete and we still have the source locator. Turtle/HTTP resume re-reads from the start and skips already accepted triples (correct; prefixes stay valid). Gzip/zstd HTTP costs another download. There is no mid-stream inflater checkpoint in v1.
unix_now
Current UNIX epoch seconds (saturating at 0 on clock skew).
window_commitment
SHA-256 over WINDOW_BYTES + window count + each hash — binds the commitment to the segmentation scheme.
write_json_atomic
Serializes value to <path>.json.tmp then renames — readers never see a partial file.