Expand description
Stage 3 — chunk: split the extracted text into structural chunks suitable for embedding and for targeted LLM extraction. Chunks follow paragraph and heading boundaries (not arbitrary character windows) and carry the running heading path so a retrieved chunk keeps its context.
Structs§
- Chunk
- One retrievable unit of a document.
Functions§
- chunk_
text - Split text into structural chunks. Heading-looking lines start a new chunk and update the heading path; paragraphs accumulate up to the target size.
- chunks_
to_ jsonl - Serialize chunks to JSON-Lines (one object per line).