Chunking is one of the most important design decisions in a retrieval-augmented generation pipeline. It is also one of the easiest to get wrong. Many systems start with fixed-size chunking because it is simple: split every document every few hundred tokens and send the chunks to an embedding model. That works for prototypes, but it often fails in production because documents are not uniform token streams. They contain sections, headings, lists, tables, code blocks and contextual dependencies. Semantic chunking keeps those relationships intact.
Semantic chunking is the process of splitting content based on meaning and structure instead of arbitrary token counts.
A semantic chunk should preserve a coherent unit of information.
That can be:
The goal is not to create equal-sized chunks.
The goal is to create useful retrieval units.
Fixed-size chunking ignores document structure.
It can split:
This creates retrieval problems.
A vector search system may retrieve a chunk that contains an answer fragment, but not the heading, assumptions or surrounding explanation needed to ground the response.
The result is often:
Raw HTML is difficult to chunk semantically because meaningful content is mixed with layout, navigation and rendering markup.
Clean Markdown is much easier to process.
Markdown exposes the structure that matters:
This makes it easier to identify natural document boundaries.
See also:
A good semantic chunking pipeline usually follows this order:
This order matters.
If you apply token splitting before cleaning and normalization, the system preserves noise and loses structure.
Useful chunks should contain more than text.
They should usually include:
This metadata improves ranking, debugging and answer attribution.
It also helps downstream systems decide which chunks should be trusted, deduplicated or refreshed.
There is no universal chunk size.
Good chunk size depends on:
As a practical starting point, many RAG systems work well with semantic chunks between 300 and 1,200 tokens.
But the boundary should be semantic first and numeric second.
A 900-token complete section is often better than three 300-token fragments that break the meaning.
Overlap can help when content depends on surrounding context.
But excessive overlap increases:
A better approach is often structural metadata plus light overlap.
For example:
This provides continuity without duplicating large amounts of text.
Technical documentation needs special handling.
A single useful chunk may need to preserve:
Splitting those apart can make retrieval much less useful.
For documentation, chunking should usually keep examples and explanations together.
This is one reason Markdown-based ingestion is especially valuable for developer docs and API references.
See:
Semantic chunking is more effective when the input is already clean.
Removing navigation, scripts, duplicated layout and irrelevant footer content reduces the amount of text that needs to be chunked in the first place.
That improves:
Token reduction should happen before chunking, not after.
See:
Semantic chunking is not just a preprocessing detail.
It directly affects retrieval quality, cost, latency and answer reliability.
The best RAG systems usually do not start by splitting raw documents into fixed token windows. They normalize content first, preserve structure, then split around meaning.
Clean Markdown makes that process simpler, cheaper and more reliable.
If you want to test how real web pages look after AI-oriented normalization, try the AI Ingestor playground.
Semantic chunking splits documents based on meaning and structure instead of fixed token counts.
For many RAG systems, yes. It usually preserves context better and improves retrieval quality.
There is no universal size. A practical starting range is often 300-1,200 tokens, but semantic boundaries should matter more than exact size.
Markdown exposes headings, lists, code blocks and tables clearly, making it easier to identify meaningful chunk boundaries.
Internal links
Chunk by meaning instead of arbitrary token windows by preserving headings, lists and tables before segmentation.
Normalize public web content into consistent, low-noise context that AI systems can index, retrieve and reason over.
Expose web content to agents and retrieval systems through a reader-style API that prioritizes clarity over browser markup.
Learn how to reduce token costs in RAG systems using content normalization, semantic chunking and Markdown-based ingestion pipelines.