Why AI makes things up about your PDF, and what actually fixes it

You upload a chapter, ask a question, and get back a paragraph that is fluent, plausible, on-topic, and not in your document. Everybody who has tried to work with an AI and a PDF has had this happen, usually at the worst possible moment — after they have already repeated it to somebody else.

It is worth understanding why, because the explanation tells you exactly which tools have fixed it and which have only made it less frequent.

The model was never obliged to look

A large language model is a very good predictor of what text should come next. When you paste a chapter into a chat window and ask a question, you have not given the model a rule. You have given it context — some words that are more available than others while it predicts.

If your chapter contains the answer, the most probable next words usually reflect it, and you get a correct answer. If your chapter does not contain the answer, the most probable next words are still a fluent answer, drawn from everything the model learned in training about that subject. Nothing in the setup distinguishes those two cases. There is no step where the model checks whether the thing it is about to say came from your file.

This is the part people get wrong when they call it a "hallucination" as though it were a malfunction. The model is doing the same thing in both cases and doing it correctly. The failure is in the arrangement, not the model.

Why bigger context windows did not fix it

The obvious response is to put more of the document in front of the model. Context windows have grown enormously, and the pitch that came with them was that you could now simply hand over the whole book.

Two things go wrong.

The first is that long context degrades in the middle. Feed a model a very long document and its ability to use material from the middle of it drops off measurably compared to the beginning and the end — the effect is well documented and shows up across model families. A 300-page book pasted into a window is not 300 pages the model can use evenly. It is a strong start, a strong finish, and a soft middle that it will still answer confidently about.

The second is that a bigger window does not add an obligation. Even with the entire document in context, nothing requires the answer to be derived from it. The model can still produce a fluent sentence from training data, and now it does so while appearing to have read everything, which is worse — you have lost the one heuristic you had, which was noticing that the document was too long to have been included.

What actually fixes it: making retrieval the only path

The fix is structural. Instead of hoping the model uses your document, you make the document the only material it has.

The shape looks like this:

1. On upload, read the document once and store it as passages. Not at question time — before any question exists.
2. On every question, search those passages first. This is a real search over your own content, returning specific passages with their locations.
3. Write the answer only from what came back. The model's job changes from "answer this question" to "answer this question using these passages", which is a different and much more constrained task.
4. If the search returns nothing relevant, say so. This is the step most implementations skip, and it is the one that matters most.

The general name for this is retrieval-augmented generation, and the important word is not retrieval, it is only. A system that retrieves passages and also permits the model to answer from memory when retrieval comes back thin has not fixed the problem. It has made it rarer and much harder to notice, which for anything you intend to rely on is a worse position than before.

The question to ask a tool

When you are evaluating anything that claims to let you chat with your documents, there is one question that separates the categories:

Can it answer when the document does not contain the answer?

Ask it something plausible and adjacent that is definitely not in your file. Something the model would obviously know from training, but that your particular document never mentions.

• If you get a confident, useful answer, the tool is using your document as context and will invent things.
• If you get "I could not find anything about that in this document", the tool is retrieval-bound, and you can rely on it.

This takes about thirty seconds and tells you more than any feature list.

Citations are the second half, and they are not the same thing

Grounding stops the answer being invented. Citations let you check that it was not. These are separate properties and it is worth being precise about which one a tool actually has.

A reference in prose — "as discussed in the section on transport" — is better than nothing, but you cannot verify it without going and looking, which is exactly the work you were trying to avoid. A citation that is a link, which opens the document at the passage the claim came from, closes the loop: reading the original sentence costs a click.

The distinction matters because prose references can themselves be wrong. A model asked to say where it found something can produce a page number the way it produces anything else. A link that resolves to a stored passage cannot, because it is not generated — it is a pointer to the retrieved chunk that was actually used.

Where chunking comes into it

There is one more failure that looks like invention and is not. If the retrieval step returns fragments, the answer is grounded in your document and still wrong, because the fragment did not contain the whole thought.

Most systems split documents into chunks of a fixed size, or one chunk per page. Both cut arguments in half. A definition that begins at the bottom of page 4 and finishes at the top of page 5 becomes two pieces, neither of which is the definition. Retrieval finds one of them, the model writes a confident answer from half a thought, and the result is faithful to a passage and unfaithful to the document.

Splitting along meaning instead — at the points where the document changes subject, wherever those fall relative to the page — is the fix, and it is why the second half of this problem is about document processing rather than about models at all.

What this means in practice

If you are choosing a tool, the checklist is short:

• Does it read the whole document on upload, or only at question time?
• Does it search your pages before every answer, with no mode that skips it?
• Does it say when it found nothing, rather than filling the gap?
• Are its citations links you can follow, or references you have to trust?
• Does it split by meaning, or by page?

None of these are things you can tell from a landing page, and all of them are things you can test in five minutes with a document you know well. That is the test worth doing before you rely on any of it.