Skip to main content

qualia_semantic_library/
lib.rs

1//! # qualia-semantic-library
2//!
3//! A Rust-native semantic library for turning a large, messy document corpus
4//! into an organised, searchable knowledge base that feeds QualiaDB.
5//!
6//! The unit of storage is the **hypermedia container** ([`container`]): a single
7//! `.hmc` ZIP file holding the original source document *and* every asset derived
8//! from it — canonical HTML, a CML/RDF semantic layer, plain text, structural
9//! chunks, and embedding vectors — behind one self-describing manifest. The
10//! original and its provenance never get separated.
11//!
12//! Pipeline (each stage is deterministic except the explicitly-marked LLM step):
13//!
14//! 1. [`ingest`] — hash + dedup a file, extract structured text/HTML, chunk it,
15//!    and pack everything into an `.hmc` container.
16//! 2. [`llm`] — an optional, swappable backend reached over HTTP (Ollama today;
17//!    a cloud API or the native QualiaDB engine later) used only for embeddings
18//!    and structured method extraction. This is the *only* external dependency
19//!    and it lives behind the [`llm::LlmBackend`] trait. It is never wired into
20//!    QualiaDB's core inference path.
21//! 3. [`library`] — an index over many containers: dedup on the embedding
22//!    manifold, route queries to the relevant region, rank by novelty.
23//! 4. [`reorganize`] — propose/apply an organised on-disk layout from container
24//!    metadata and clusters.
25//!
26//! This crate is an **offline developer tool**. It honours the project rule that
27//! external LLM runtimes may be used as independent tools but are never compiled
28//! into the QualiaDB runtime — hence the HTTP seam and the `llm-http` feature.
29
30pub mod container;
31pub mod embedding;
32
33#[cfg(feature = "pdf")]
34pub mod ingest;
35
36#[cfg(feature = "llm-http")]
37pub mod llm;
38
39pub mod library;
40pub mod ontology;
41pub mod reorganize;
42
43pub use container::{
44    AssetEntry, AssetKind, HmcContainer, HmcError, HmcManifest, HmcWriter, SourceInfo,
45    HMC_EXTENSION, HMC_FORMAT_VERSION,
46};