Skip to main content

Module tokenize

Module tokenize 

Source
Expand description

UTF-8 tokenizer + sentence split. Offsets are byte spans.

Structs§

Sentence
Token

Enums§

TokenKind

Functions§

split_sentences
Splits text into sentences.
tokenize
?::tokenize.