Chunk at semantic boundaries
Good defaults are heading changes, list boundaries and section-level code examples. Splitting in the middle of those structures usually harms retrieval quality.
Technical blogTechnical article
A practical semantic chunking guide
Semantic chunking is mostly about respecting document structure. If headings, paragraphs and code examples survive extraction, chunking becomes a controlled operation instead of a repair job.
Primary use
How to build chunk boundaries around document meaning instead of arbitrary token slices.
Recommended flow
Fetch, clean, measure tokens, then hand consistent Markdown to agents or retrieval systems.
Next step
Use the Playground to compare raw HTML against optimized output before integrating the API.
Avoid overfitting to one token limit
A chunker should adapt to content shape, not force every section into the same size bucket. Clean extraction gives that flexibility without losing topical coherence.
Internal links
Related technical paths
Chunk quality
Semantic chunking starts with clean structure
Chunk by meaning instead of arbitrary token windows by preserving headings, lists and tables before segmentation.
Token efficiency
Context optimization for LLM retrieval
Reduce noisy HTML and preserve semantic structure so every prompt and chunk carries more useful signal per token.
Agent access
Reader API for AI systems
Expose web content to agents and retrieval systems through a reader-style API that prioritizes clarity over browser markup.