Skip to main content

Module chunk

Module chunk 

Source
Expand description

Stage 3 — chunk: split the extracted text into structural chunks suitable for embedding and for targeted LLM extraction. Chunks follow paragraph and heading boundaries (not arbitrary character windows) and carry the running heading path so a retrieved chunk keeps its context.

Structs§

Chunk
One retrievable unit of a document.

Functions§

chunk_text
Split text into structural chunks. Heading-looking lines start a new chunk and update the heading path; paragraphs accumulate up to the target size.
chunks_to_jsonl
Serialize chunks to JSON-Lines (one object per line).