Expand description
Document NLP (tokenize, gazetteer, span plans). Engine capability, not Vibe. Document NLP in the engine: bytes → tokens → gazetteer → span plans.
Lives next to text_span and lexicon. Not VibeScript. Not an MCP-only tool.
Year-one is tokenize + gazetteer + spans. Not FrameNet, RST, OpenIE, NLI, or MT.
Re-exports§
pub use capability::lookup;pub use capability::NlpCapability;pub use capability::NLP_CAPABILITIES;pub use contracts::analyze_document_into;pub use contracts::required_capacities;pub use contracts::slice_source;pub use contracts::AnalysisState;pub use contracts::BufferChannel;pub use contracts::DocumentCapacities;pub use contracts::DocumentSummary;pub use contracts::DocumentViewBuffers;pub use contracts::HitView;pub use contracts::NlpContractError;pub use contracts::NormView;pub use contracts::PlanView;pub use contracts::SentenceView;pub use contracts::TokenView;pub use contracts::ANNOTATION_CONTRACT_VERSION;pub use contracts::NORM_INDEX_NONE;pub use contracts::SENTENCE_ID_NONE;
Modules§
- budget
- Shared NLP host resource budgets.
- capability
- Honest maturity metadata for public
NLP.*invoke IDs. - contracts
- Versioned annotation contracts and caller-buffered document views (NLP-100).
- coref
- Bounded coreference: exact-string grouping plus an experimental pronoun sieve.
- emit
- Span-annotation plans. Host seals Quins via
text_span::annotation_quin. - frame
- Frame semantics — basic frame element identification.
- fst
- Finite-State Transducer for morphology.
- gazetteer
- Aho-Corasick gazetteer over interned lexicon surfaces.
- graphrag
- Graph-augmented retrieval — keyword-based triple retrieval.
- hash
- Content hash for spans — same 60-bit FNV-1a as
lexicon::generate_60bit_token. - link
- Link only to known lexicon / ontology IDs.
- normalize
- Narrow date / number normalizers. Extended with QUDT unit vocabulary.
- relation
- Relation extraction — RDF-Star triple extraction via shallow patterns.
- span
- UTF-8 byte span. Same fields as core-db
TextSpanminus the hash (computed at emit). - substrate
- Full symbolic pipeline — orchestrates the NLP modules end-to-end.
- terms
- Interned gazetteer lexicon. Hits only emit these IRIs.
- tokenize
- UTF-8 tokenizer + sentence split. Offsets are byte spans.
Structs§
- Document
Analysis - One-shot document analysis. Desktop and Vibe hosts call this; they do not reimplement tokenisation.
Functions§
- analyze_
document - Analyzes a document into annotations.