Skip to main content

Module ingest

Module ingest 

Source
Expand description

Ingest orchestration: a single file → a .hmc hypermedia container.

This whole module is deterministic — no LLM. It acquires the source, extracts structured text/HTML, chunks it, emits the CML annotation layer, and packs the original plus every derived asset into one container. Embeddings and LLM method-extraction are separate enrichment passes (crate::llm) so ingest stays reproducible and offline-clean.

Modules§

acquire
Stage 1 — acquire: read a file, establish its identity (BLAKE3), and detect its source kind. No extraction yet; this stage only produces the verbatim bytes + a SourceInfo the rest of the pipeline keys on.
chunk
Stage 3 — chunk: split the extracted text into structural chunks suitable for embedding and for targeted LLM extraction. Chunks follow paragraph and heading boundaries (not arbitrary character windows) and carry the running heading path so a retrieved chunk keeps its context.
cml
Stage 4 — CML layer: emit a small RDF/Turtle annotation over the document that QualiaDB can ingest. This is the semantic layer on top of the HTML: the document as a node, its sections, and heuristically-detected formal method candidates (Definition / Theorem / Lemma / Algorithm / Proposition), each carrying provenance back to the source.
extract
Stage 2 — extract: turn the verbatim source into a canonical structured representation: plain text (the cheap full-text surface) and canonical HTML (the structure-bearing surface that the CML layer annotates).

Structs§

IngestOptions
Options controlling an ingest run.
IngestResult
Outcome of ingesting one file.

Functions§

ingest_file
Ingest a single file into a container. Returns the container path.
ingest_path
Ingest a file or a directory tree. Returns per-file results.