Chunking comes down to three decisions: where to cut, how big the pieces are, and how much context each piece still carries once it's on its own. Most engineering effort goes into the first one — semantic splitters, LLM-directed splitters — and that's the decision with the smallest marginal effect on retrieval. The second and third barely get measured at all, yet they determine how much of your context window is filler. Two pieces of evidence that look contradictory actually agree: Chroma's technical report Evaluating Chunking Strategies for Retrieval, published 3 July 2024, found that chunking strategy can move recall by up to 9 percentage points; Is Semantic Chunking Worth the Computational Cost? (Qu, Tu, Bao — Findings of NAACL 2025, pp. 2155–2177) found that the compute cost of semantic chunking is not justified by consistent gains. Both are right. The spread comes overwhelmingly from chunk size and overlap, not from semantics.

Three splitters, three distinct failure modes

Fixed-window token splitting. Count N tokens, cut. It severs sentences and sometimes words, leaving a definition split across two chunks where neither half retrieves. What you get in exchange is the only property that is genuinely reliable: an exact token budget. You know precisely how many tokens enter the prompt after top-k, which matters when you're costing a pipeline or deciding what to truncate.

Recursive separator splitting. Walk a priority list of separators, dropping to the next when chunks are still too large. LangChain's RecursiveCharacterTextSplitter defaults to ["\n\n", "\n", " ", ""]; the Chroma report used an extended list including sentence punctuation, ["\n\n", "\n", ".", "?", "!", " ", ""]. The failure mode here is quiet: degradation is silent. There is no separator hit-rate exposed anywhere. Feed it text extracted from a PDF with no blank lines and long unpunctuated runs, and it falls all the way through to splitting on spaces — functionally identical to fixed-window — and nothing in your logs says so. If you run recursive splitting in production, instrument which separator level actually fired, per document. That one counter catches corpus-quality regressions that no retrieval metric surfaces until much later.

Semantic splitting. Embed groups of adjacent sentences, measure cosine distance between consecutive groups, cut where distance spikes. LlamaIndex's SemanticSplitterNodeParser defaults to buffer_size=1 and breakpoint_percentile_threshold=95. langchain_experimental's SemanticChunker offers four threshold modes with defaults percentile=95, standard_deviation=3, interquartile=1.5, and gradient=95, where the gradient mode applies the percentile to the derivative of the distance series rather than the raw distances.

That percentile default deserves to be stated plainly, because it is the thing people miss: the threshold is relative to the distance distribution of the document you happen to be splitting. A tightly focused API reference and a rambling meeting transcript will both yield breakpoints at roughly 5% of positions. The splitter is not finding topic boundaries in any absolute sense; it is finding the top 5% of whatever variation exists in that particular document. Chunk length is entirely uncontrolled as a result, and the two parameters that would constrain it — min_chunk_size and number_of_chunks — both default to None.

The defaults are the biggest trap

These are from source, not documentation:

  • langchain_text_splitters' base TextSplitter defaults to chunk_size=4000, chunk_overlap=200, with length_function=len. That 4000 counts characters, not tokens. In English that's roughly 1000 tokens; in Chinese or Japanese it's 2000+. The same config produces chunks that differ by a factor of two across a multilingual corpus, and embedding quality does not degrade linearly with length. The constructor validates that chunk_overlap <= chunk_size and raises ValueError otherwise — it does not, and cannot, warn you that your unit is wrong.
  • LlamaIndex's llama_index.core.constants defines DEFAULT_CHUNK_SIZE = 1024 and DEFAULT_CHUNK_OVERLAP = 20. But SentenceSplitter uses a separate constant, SENTENCE_CHUNK_OVERLAP = 200. Two defaults in one library, a factor of ten apart, selected by which class you happened to instantiate.
  • SentenceSplitter's paragraph_separator defaults to "\n\n\n" — three newlines. Markdown separates paragraphs with two. On any Markdown corpus that separator effectively never fires, and the real work falls to secondary_chunking_regex.

The Chroma report names the widely copied 800/400 configuration directly — it is the default chunking strategy documented for OpenAI's vector stores. With text-embedding-3-large and n=5 retrieved chunks: 87.9% recall, but 1.4% precision and 1.4% IoU.

No framework default was tuned on your corpus. Treat every one of them as a starting coordinate, never an answer.

What size and overlap actually do

Representative numbers from the Chroma report, all at the same setting (text-embedding-3-large, n=5):

Strategy Size Overlap Recall Precision IoU
Recursive 200 0 88.1% 7.0% 6.9%
ClusterSemantic 200 0 87.3% 8.0% 8.0%
LLMSemantic variable 0 91.9% 3.9% 3.9%
TokenText 800 400 87.9% 1.4% 1.4%

The metrics are token-level. Recall is the proportion of relevant tokens retrieved; precision is the proportion of retrieved tokens that are relevant; IoU is the Jaccard-style ratio of the two. This is a far more honest accounting than chunk-level hit rate, which scores an 800-token chunk containing 30 useful tokens as a perfect hit. If your current evaluation reports chunk-level hit rate (see RAG Retrieval Evaluation for the per-query metrics worth tracking instead), you are structurally blind to the cost side of large chunks — every chunking decision will look neutral, because the only thing you measure is the side where they all tie.

Three things to read off that table.

Recall differences are smaller than the discourse suggests. All four strategies sit between 87% and 92%. If recall is the only number you track, chunking looks like a decision that barely matters — which is exactly why so many teams conclude it doesn't and move on to reranking.

Precision and IoU differ by more than 5x. This is where chunking actually lives. It sets how much noise the model has to read past, how much of your context budget is spent on tokens nobody needed, and — once you're paying per token at production volume — a meaningful share of your bill. A 200/0 recursive configuration and an 800/400 token configuration retrieve roughly the same relevant material; one of them ships five times more junk alongside it.

Overlap is close to pure loss. At 800/400 every span of text is stored twice, and a top-5 retrieval can return nearly half its tokens as duplicated content — content that also displaces genuinely new material from the same k slots. The report states the overlap half of this explicitly: reducing chunk overlap improves IoU, because the metric penalizes redundant information. The advantage of smaller chunks (200 tokens) on precision and IoU is read off the table above; the report itself stops short of prescribing a configuration.

The usual defense of overlap is that it prevents a key sentence from being severed at a boundary. That defense is real but the remedy is badly targeted: you are paying a global 2x storage-and-retrieval cost to patch a local boundary problem. Fix the boundaries instead (structure-aware splitting), or make each chunk self-sufficient (next section). If you keep overlap at all, keep it small — 10–15% of chunk size, not 50% — and make it earn its place in an experiment rather than inheriting it from a tutorial.

When semantic chunking earns its compute

The NAACL 2025 Findings paper tested semantic chunking against simple fixed-size chunking on document retrieval, evidence retrieval, and retrieval-based answer generation, and could not establish consistent gains. That does not contradict Chroma, and understanding why matters for your decision.

The two strongest performers in the Chroma table are not naive semantic chunkers. ClusterSemanticChunker uses dynamic programming to maximize within-chunk similarity subject to a target length, and LLMSemanticChunker asks a model to predict split points directly. Both differ fundamentally from "embed sentence groups, cut at the 95th percentile of cosine distance" — and that naive variant is precisely what ships as the reachable default in the mainstream frameworks.

So the position is: don't use percentile-threshold semantic chunking. It costs a full extra embedding pass over your corpus, produces chunks of uncontrolled length, and the published evidence does not support the gain.

What you should do first is structure-aware splitting: Markdown heading hierarchy, HTML semantic elements, AST boundaries in code, clause numbers in contracts and regulations, speaker turns in transcripts. Those boundaries were placed by a human who understood the document. They are strictly better than boundaries guessed from cosine distance, and they cost nothing. Semantic chunking only becomes a serious candidate when the document genuinely has no structural markup at all — raw OCR output, unformatted speech transcripts — and even then it should enter as one arm of an experiment, not as the default.

Two interventions with better returns than tuning size

Give every chunk its context back. Anthropic's Contextual Retrieval, published 19 September 2024, prepends a short generated explanation to each chunk before indexing. The prompt passes the whole document alongside the chunk and asks for a short succinct context situating it within the document, for retrieval purposes. The reported top-20 retrieval failure rates: 5.7% baseline, 3.7% with contextual embeddings (−35%), 2.9% adding contextual BM25 (−49%; on fusing the vector and text arms at the database layer, see MongoDB Hybrid Search), 1.9% adding reranking (−67%). Cost is 50–100 extra tokens per chunk, plus a one-time generation pass. Compare that to the one or two points you might squeeze out of a chunk-size sweep. Anthropic's own guidance on boundaries stays deliberately unspecific — it notes that size, boundary, and overlap all affect performance without prescribing values, which is the honest position given how corpus-dependent the answer is.

Late chunking. arXiv:2409.04701 (Günther, Mohr, Williams, Wang, Xiao; submitted 7 September 2024, v3 dated 7 July 2025) runs the entire long document through the transformer first and applies chunking after the transformer and just before mean pooling — hence "late". Each chunk embedding then carries full-document context, with no additional training required. The hard constraint is easy to miss and disqualifying: it needs a long-context embedding model that uses mean pooling. Models that produce their sentence representation from a CLS token never aggregate per-token representations in the first place, so there is nothing to pool segment-wise, and the technique simply does not apply. Check your model's pooling strategy before you plan around this.

Both interventions share a property that explains why they beat splitter tuning: they don't move the boundaries, they change how much each chunk can see when it is embedded. That is the actual problem naive semantic chunking is groping at and failing to solve.

Turning parameter selection into a reproducible experiment

Do not pick chunking parameters using a public benchmark. Chroma's Generative Benchmarking report (Kelly Hong, Anton Troynikov, Jeff Huber, with Morgan McGuire of Weights & Biases, 7 April 2025) is the cleanest available demonstration of why. jina-embeddings-v3 consistently outperforms text-embedding-3-large on MTEB tasks, yet performs worse on Weights & Biases' actual production support data. Public benchmarks are generically domain-scoped, unusually clean, and increasingly contaminated by training exposure.

Their pipeline is two steps. First, filter documents with an LLM judge aligned against human relevance judgments — for the WandBot corpus this reduced 13,319 documents to 8,490 suitable for query generation. Second, generate queries with domain context and real example queries as steering signals rather than naive prompting. The difference is measurable: queries generated with context and examples produced a query-document cosine similarity distribution with KL divergence 0.159 against real production queries, versus 0.207 for naive generation. Their conclusion belongs above every retrieval dashboard: strong public-benchmark performance for an embedding model does not guarantee comparable performance in your pipeline.

Annotate at token level. The chunking_evaluation package that accompanies the Chroma report installs directly from GitHub (pip install git+https://github.com/brandonstarxel/chunking_evaluation.git). GeneralEvaluation.run(chunker, embedding_function) returns IoU and recall as mean and standard deviation; custom splitters plug in by subclassing BaseChunker. It also ships a SyntheticEvaluation pipeline that generates queries and matching source excerpts from your own corpus, then filters weak excerpts and duplicates. Dependencies include tiktoken, chromadb, and the OpenAI and Anthropic clients, so budget for API cost.

Keep the grid orthogonal. Move one dimension at a time and freeze everything else:

# Frozen: embedding model, top-k, reranker, tokenizer, random seed
GRID = [(200, 0), (400, 0), (400, 60), (600, 0), (800, 0), (800, 400)]

rows = []
for size, overlap in GRID:
    chunker = MyChunker(chunk_size=size, chunk_overlap=overlap)  # subclasses BaseChunker
    r = evaluation.run(chunker, embedding_function)
    rows.append({
        "size": size, "overlap": overlap,
        "recall": r["recall_mean"], "recall_std": r["recall_std"],
        "iou": r["iou_mean"], "iou_std": r["iou_std"],
    })

The single most common mistake in this space is changing the splitter and the embedding model in the same run and attributing the delta to the splitter.

Report variance and run paired tests. Chroma reports standard deviations for a reason: per-query variance is typically far larger than the gap between configurations. A one-point difference in mean recall with a 15-point per-query standard deviation is noise, and it will not survive contact with next quarter's corpus. Use ir-measures or pytrec_eval for per-query scores and a paired test — pytrec_eval ships an example that compares two runs with a paired Student's t-test via scipy.stats.ttest_rel. Ranking configurations by mean alone is how teams ship a change that was a coin flip.

Record three pieces of metadata in every results row: the splitter implementation and version, the tokenizer, and the embedding model version. chunk_size=400 under a character-counting implementation and a token-counting one are different experiments producing incomparable numbers, and six months later nobody will be able to tell them apart from the results table alone.

A defensible starting configuration

Structure-aware recursive splitting where markup exists (Markdown headings, HTML, AST boundaries), targeting 400–600 tokens with zero overlap, a generated contextual prefix on every chunk, retrieval at top-k 10 reranked down to 3–5 (what a larger top-k costs you depends on index parameters — see Tuning HNSW). Every element of that is supported by the evidence above rather than by convention.

Then run three experiments in order. Sweep chunk size first (200/400/600/800, overlap fixed at 0) — this is where the precision differences live. Second, test whether overlap buys anything above noise on your corpus; expect it not to, and delete it when it doesn't. Only third, evaluate late chunking or a length-constrained semantic splitter as a candidate, with the token-level metrics and paired significance test already in place from steps one and two.

Measure first, then tune. Chunk size and overlap are things you can measure; semantic chunking is mostly a thing people guess at.