Skip to main content

Module nlp

Module nlp 

Source
Expand description

Document NLP (tokenize, gazetteer, span plans). Engine capability, not Vibe. Document NLP in the engine: bytes → tokens → gazetteer → span plans.

Lives next to text_span and lexicon. Not VibeScript. Not an MCP-only tool. Year-one is tokenize + gazetteer + spans. Not FrameNet, RST, OpenIE, NLI, or MT.

Re-exports§

pub use capability::lookup;
pub use capability::NlpCapability;
pub use capability::NLP_CAPABILITIES;
pub use contracts::analyze_document_into;
pub use contracts::required_capacities;
pub use contracts::slice_source;
pub use contracts::AnalysisState;
pub use contracts::BufferChannel;
pub use contracts::DocumentCapacities;
pub use contracts::DocumentSummary;
pub use contracts::DocumentViewBuffers;
pub use contracts::HitView;
pub use contracts::NlpContractError;
pub use contracts::NormView;
pub use contracts::PlanView;
pub use contracts::SentenceView;
pub use contracts::TokenView;
pub use contracts::ANNOTATION_CONTRACT_VERSION;
pub use contracts::NORM_INDEX_NONE;
pub use contracts::SENTENCE_ID_NONE;

Modules§

budget
Shared NLP host resource budgets.
capability
Honest maturity metadata for public NLP.* invoke IDs.
contracts
Versioned annotation contracts and caller-buffered document views (NLP-100).
coref
Bounded coreference: exact-string grouping plus an experimental pronoun sieve.
emit
Span-annotation plans. Host seals Quins via text_span::annotation_quin.
frame
Frame semantics — basic frame element identification.
fst
Finite-State Transducer for morphology.
gazetteer
Aho-Corasick gazetteer over interned lexicon surfaces.
graphrag
Graph-augmented retrieval — keyword-based triple retrieval.
hash
Content hash for spans — same 60-bit FNV-1a as lexicon::generate_60bit_token.
link
Link only to known lexicon / ontology IDs.
normalize
Narrow date / number normalizers. Extended with QUDT unit vocabulary.
relation
Relation extraction — RDF-Star triple extraction via shallow patterns.
span
UTF-8 byte span. Same fields as core-db TextSpan minus the hash (computed at emit).
substrate
Full symbolic pipeline — orchestrates the NLP modules end-to-end.
terms
Interned gazetteer lexicon. Hits only emit these IRIs.
tokenize
UTF-8 tokenizer + sentence split. Offsets are byte spans.

Structs§

DocumentAnalysis
One-shot document analysis. Desktop and Vibe hosts call this; they do not reimplement tokenisation.

Functions§

analyze_document
Analyzes a document into annotations.