Expand description
Ingest orchestration: a single file → a .hmc hypermedia container.
This whole module is deterministic — no LLM. It acquires the source,
extracts structured text/HTML, chunks it, emits the CML annotation layer, and
packs the original plus every derived asset into one container. Embeddings
and LLM method-extraction are separate enrichment passes (crate::llm) so
ingest stays reproducible and offline-clean.
Modules§
- acquire
- Stage 1 — acquire: read a file, establish its identity (BLAKE3), and detect
its source kind. No extraction yet; this stage only produces the verbatim
bytes + a
SourceInfothe rest of the pipeline keys on. - chunk
- Stage 3 — chunk: split the extracted text into structural chunks suitable for embedding and for targeted LLM extraction. Chunks follow paragraph and heading boundaries (not arbitrary character windows) and carry the running heading path so a retrieved chunk keeps its context.
- cml
- Stage 4 — CML layer: emit a small RDF/Turtle annotation over the document that QualiaDB can ingest. This is the semantic layer on top of the HTML: the document as a node, its sections, and heuristically-detected formal method candidates (Definition / Theorem / Lemma / Algorithm / Proposition), each carrying provenance back to the source.
- extract
- Stage 2 — extract: turn the verbatim source into a canonical structured representation: plain text (the cheap full-text surface) and canonical HTML (the structure-bearing surface that the CML layer annotates).
Structs§
- Ingest
Options - Options controlling an ingest run.
- Ingest
Result - Outcome of ingesting one file.
Functions§
- ingest_
file - Ingest a single file into a container. Returns the container path.
- ingest_
path - Ingest a file or a directory tree. Returns per-file results.