Skip to main content
Module tokenize
qualia_
core_
db
0.0.41
Module tokenize
Module Items
Structs
Enums
Functions
In qualia_
core_
db::
nlp
qualia_core_db
::
nlp
Module
tokenize
Copy item path
Source
Expand description
UTF-8 tokenizer + sentence split. Offsets are byte spans.
Structs
§
Sentence
Token
Enums
§
Token
Kind
Functions
§
split_
sentences
Splits
text
into sentences.
tokenize
?::tokenize
.