Expand description
SPARQL tokenizer.
Turns a SPARQL fragment (a FILTER/BIND expression, or a group graph pattern)
into a flat Vec<Token> that the expression and pattern parsers consume by
recursive descent. This is the front-end the old string-slicing parser never
had — the AST, planner, and executor already support the full algebra, so a
real tokenizer + parser is all that stands between a query string and the
engine.
Not zero-heap: parsing is a cold, one-shot path per query (unlike execution),
so a Vec<Token> + String interning is the right trade for correctness and
clarity. Execution stays on the zero-heap SparqlQueryContext arenas.
Enums§
- Token
- A lexical token.
Functions§
- tokenize
- Tokenize a SPARQL fragment. Returns an error string on a malformed literal or an unterminated IRI/string.