Chunking documents for AI: page breaks are the wrong place to split

If you are building anything that answers questions from documents, the chunking decision will do more damage than your choice of model. It is also the part that gets the least attention, because it looks like plumbing.

Here is the failure it produces, which is worth recognising before the explanation: an answer that is genuinely grounded in a retrieved passage, cites that passage correctly, and is wrong about the document. No hallucination is involved. The system did what it was told. It was told to split the document in a place that destroyed the meaning.

The two defaults, and what they break

Almost every quick implementation uses one of two splits.

Fixed-size chunks. Cut every N characters or tokens, usually with some overlap. It is trivial to implement, it produces uniform chunks that are pleasant to embed, and it is indifferent to what the document is saying. A 512-token boundary lands wherever it lands — mid-sentence, mid-definition, between a claim and the "because" that supports it.

One chunk per page. More intuitive and worse in a specific way. Page boundaries in a PDF are a fact about typesetting, not about content. A page break is placed by a layout engine to fill a rectangle. Treating it as a semantic boundary is treating the width of the paper as an argument about meaning.

Both produce the same class of bug. Consider a definition that begins at the bottom of page 4 and completes at the top of page 5:

...we define the transport coefficient as the ratio of flux to

— page break —

the gradient that drives it, measured under steady-state conditions.

Split at the page, you now have two chunks. The first contains the term and half the definition. The second contains the other half and no term at all.

Ask "what is the transport coefficient?" Retrieval scores the first chunk highly — the term is right there — and returns it. The model dutifully writes an answer from what it was given: the transport coefficient is the ratio of flux to something. The steady-state condition, which is the part that makes the definition correct, is in a chunk that never got retrieved because it does not contain the term.

The answer is grounded. The citation is accurate. The definition is wrong. And nothing in the pipeline can detect this, because every component did its job.

Overlap does not solve it, it moves it

The standard patch is overlapping windows: each chunk repeats the last N tokens of the previous one, so a thought cut in half appears whole somewhere.

It helps and it is not a fix, for three reasons.

It is a guess about length. Overlap of 50 tokens saves a split definition and loses a split argument that ran 400 tokens. You are betting that the important units are shorter than your overlap, and in dense material they are not.

It corrupts retrieval scoring. Duplicated text means several chunks contain the same passage, so a query matching that passage retrieves near-identical results. Your top-k is now three copies of one thing and the diversity you needed is pushed off the list.

It inflates the index. 25% overlap is 25% more vectors, more storage, more query cost, forever, in exchange for a partial mitigation.

Splitting along meaning

The alternative is to put boundaries where the document changes subject, wherever those fall relative to the pages.

In practice that means using the document's own structure — headings, sections, list boundaries, the shift from prose to a worked example — and where there is no explicit structure, detecting topic shifts from the text itself. The units come out uneven, which is the point: a tightly argued definition is short and a discursive introduction is long, and forcing both to 512 tokens serves neither.

Three things fall out of doing this that are worth more than the retrieval improvement itself:

Chunks become citable. A chunk that is a whole thought maps to a place in the document a human recognises — "the definition of the transport coefficient, section 3.2, pages 4-5". A fixed-size chunk maps to "characters 8,192 to 8,704", which cannot be shown to anybody.

Retrieval gets more precise, not just more accurate. Whole-thought chunks embed to a coherent point in vector space. A chunk containing the end of one argument and the start of another embeds to a point between two topics and is mildly relevant to everything, which is the worst possible property for a retrieval unit.

Structure becomes navigable. Once you have extracted the document's real sections and the page ranges they cover, "the bit about transport" is somewhere a user can be sent, rather than a search they have to run.

The cost, honestly

Semantic chunking is more expensive at ingest. You have to parse the document properly, and detecting topic boundaries in unstructured prose usually means a model pass over the text — which is real money on a 3,000-page document and real latency.

That is the trade, and it is a good one for a specific reason: ingest happens once, retrieval happens forever. A document is chunked one time and queried for as long as somebody keeps it. Paying once at upload to make every subsequent answer better is close to the best ratio available in this kind of system.

It does mean upload cannot be synchronous. A proper read of a long document takes minutes, not seconds, which forces the ingest into a background job with progress the user can see — a constraint on your architecture that arrives from a decision that looked like plumbing.

Where scanned pages fit

Everything above assumes there is text to split. A scanned chapter or a photographed handout has no text layer, and a naive pipeline extracts nothing, indexes nothing, and then answers questions about the document as though those pages did not exist — silently, with no error, which is the worst way for it to fail.

Transcription has to happen before chunking, and it has to be part of the same ingest rather than a separate feature a user has to know to switch on. A pipeline where OCR is opt-in produces documents that are half-indexed and gives no sign of it.

A checklist

If you are building this, the questions worth answering explicitly:

• Where do your boundaries come from? If the answer is a number, you have the bug above.
• Can a chunk be described to a human? If not, you cannot cite it.
• What happens to a thought that spans a page break? Trace one through your pipeline by hand. It is a twenty-minute exercise and it is usually alarming.
• What happens to a page with no text layer? If the answer is "nothing", find out how often that occurs in your real corpus.
• Does retrieval failure produce a refusal? If thin retrieval falls through to the model's own knowledge, none of the above matters — you have a system that is right most of the time and cannot tell you when it is not.

That last one is the one that determines whether anybody can rely on the thing. The rest determines how often they are pleased with it.