Expand description
§qualia-semantic-library
A Rust-native semantic library for turning a large, messy document corpus into an organised, searchable knowledge base that feeds QualiaDB.
The unit of storage is the hypermedia container (container): a single
.hmc ZIP file holding the original source document and every asset derived
from it — canonical HTML, a CML/RDF semantic layer, plain text, structural
chunks, and embedding vectors — behind one self-describing manifest. The
original and its provenance never get separated.
Pipeline (each stage is deterministic except the explicitly-marked LLM step):
ingest— hash + dedup a file, extract structured text/HTML, chunk it, and pack everything into an.hmccontainer.llm— an optional, swappable backend reached over HTTP (Ollama today; a cloud API or the native QualiaDB engine later) used only for embeddings and structured method extraction. This is the only external dependency and it lives behind thellm::LlmBackendtrait. It is never wired into QualiaDB’s core inference path.library— an index over many containers: dedup on the embedding manifold, route queries to the relevant region, rank by novelty.reorganize— propose/apply an organised on-disk layout from container metadata and clusters.
This crate is an offline developer tool. It honours the project rule that
external LLM runtimes may be used as independent tools but are never compiled
into the QualiaDB runtime — hence the HTTP seam and the llm-http feature.
Re-exports§
pub use container::AssetEntry;pub use container::AssetKind;pub use container::HmcContainer;pub use container::HmcError;pub use container::HmcManifest;pub use container::HmcWriter;pub use container::SourceInfo;pub use container::HMC_EXTENSION;pub use container::HMC_FORMAT_VERSION;
Modules§
- container
- The
.hmchypermedia container: a single, fast-compression (ZIP) file that bundles an original source document together with all of its derived assets (canonical HTML, CML/RDF, plain text, structural chunks, embeddings, extraction metadata) behind one self-describingHmcManifest. - embedding
- Embedding storage codec + similarity. Always compiled (the library indexes and searches vectors even when built without the HTTP LLM backend).
- ingest
- Ingest orchestration: a single file → a
.hmchypermedia container. - library
- The semantic library: an index over a directory of
.hmccontainers. - llm
- The external-LLM seam. Embeddings and structured method-extraction are the only steps that need a language model; everything else in the crate is deterministic. That model is reached over HTTP behind this trait, so it can be a local Ollama server today, a hosted API tomorrow, or the native QualiaDB engine later — without touching the rest of the pipeline. It is never compiled into QualiaDB’s core inference path.
- ontology
- reorganize
- Reorganise the library: derive a clean, browsable on-disk layout from
container metadata. Containers are content-addressed (
<doc_id>.hmc), which is great for dedup but unreadable for a human. This produces a parallel, human-facing tree organised by topic/tag with descriptive filenames, while the content-addressed store stays the source of truth.