This page surveys how text is split before it is embedded and indexed in a retrieval pipeline. It opens with why chunking matters, since the split affects both what the retriever finds and how good the answer is. The main body describes five approaches in turn, fixed size, recursive, document based, semantic, and agentic, and each entry covers the idea and where it fits. Semantic chunking gets extra detail on its steps and on the buffer size and threshold parameters that control merging. Two shorter sections cover proposition based retrieval and ChunkFormer, a masked chunking model for long form speech transcription trained on Vietnamese data. Links to the RAG overview and fine tuning notes close the page.
The Role of Chunking
- Splitting text (chunking) is an important step in a RAG system for optimizing how information is retrieved and used
- It directly affects the quality of retrieval results and of the model's responses
Types of Chunking
Fixed Size Chunking
- Splits text into segments of a fixed size based on character count
- Simple, but can cut across the meaning of a sentence or paragraph
Recursive Chunking
- Splits text based on structure such as line breaks, then applies Fixed Size Chunking
- Respects the natural structure of the text better
Document-Based Chunking
- Splits a document based on its inherent structure (headings, sections, chapters)
- Suited to documents with clear structure
Semantic Chunking
- Splits text by semantic meaning, grouping sentences or paragraphs that are semantically related
- How it works:
- Split the document into small segments
- Embed the segments into vectors
- Compare the similarity between consecutive segments
- Merge segments with high similarity (above a certain threshold) to form chunks that carry their own meaning
- Key parameters:
buffer_size: The number of sentences grouped together before semantic chunking is performedthreshold: The similarity threshold that decides whether segments are merged
- More effective at producing chunks with complete meaning
Agentic Chunking
- Uses an LLM to split documents automatically
- The LLM can understand context and complex structure to produce optimal chunks
Proposition-Based Retrieval
- An advanced RAG method that focuses on retrieving propositions rather than large blocks of text
- Improves RAG performance by retrieving more precise and relevant information
ChunkFormer
ChunkFormer: Masked Chunking Conformer for long-form speech transcription (ICASSP 2025)
- Long-form transcription: processes up to 16 hours of audio with a 110M-parameter model on an 80GB GPU (4 hours on a 24GB GPU)
- Endless decoding: processes long audio on memory-limited GPUs without losing context (history and future context) thanks to streaming
- Masked Batching Technique: removes padding in batches, optimizing processing of audio with varying lengths
- Model trained on 3000h of public Vietnamese data: HuggingFace | GitHub | Paper #ASR #speech2text #long-form #Vietnamese #ZaloAI
See also: RAG Overview · RAG Fine Tuning