Expand description
CST trivia — comments, whitespace, and exact token text (T36).
The AST drops comments and whitespace. The CST preserves them so
that poet translate and poet format can round-trip source text
without destroying commentary.
§Design
Triviarepresents a comment or whitespace gap.TriviaSinkcollects trivia as the lexer advances.CstNodewraps an AST node with its leading and trailing trivia.
The trivia sink is optional — the lexer can run in “AST mode” (trivia discarded, current behaviour) or “CST mode” (trivia collected). This avoids changing existing behaviour.
Reference: docs/vibescript-full-impl-PLAN.md §8.8 T36.
Structs§
- CstNode
- A CST node — an AST node with its leading and trailing trivia.
- Trivia
- A single trivia element — a comment or whitespace gap.
- Trivia
Sink - A sink that collects trivia as the lexer advances.
Use
TriviaSink::newto create a collecting sink, orTriviaSink::disabledto create a discarding sink.
Enums§
- Trivia
Kind - The kind of trivia.
Functions§
- extract_
comments - Extract trivia from source text by scanning for comments.