KHAGAN AYDIN
Essays

Why Language Models Make Things Up, and How to Check Their Answers

A fluent answer is not a verified answer. Why invented facts happen, and a routine for catching them.

· 5 min read

Last month a federal judge in New York fined two lawyers and their firm a combined $5,000. In the case Mata v. Avianca, they had filed a brief that cited six court decisions. None of the six existed. ChatGPT had produced them, complete with case names, quotations and page numbers, and when one of the lawyers asked whether the cases were real, it said they were.

The lawyers’ mistake was not using the tool. It was treating a fluent answer as a checked one. That mistake is now being made every day, in every industry, mostly without a judge to catch it.

Why it happens

A language model is trained to predict the next words in a text. It learns patterns from a very large amount of writing: how legal citations look, how studies are described, how numbers appear in reports. It does not look facts up in a database, and it has no built-in signal that says “I do not actually know this”.

So when a question asks for something the model has not seen often, it does what it was trained to do. It produces text that looks like a correct answer. A court citation in the right format. A statistic with a believable decimal. A quote in the author’s style. The output is fluent because fluency is what the model is good at. Accuracy is a separate property, and nothing in the text tells you which one you got.

Where to expect it

Invented content is not spread evenly. It concentrates in a few places:

  • References. Court cases, papers, book titles, links. Links and paper identifiers are invented especially often.
  • Exact quotes. The model reproduces the style of a person, not their actual words.
  • Numbers. Statistics, dates, prices and calculations, especially in long chains of reasoning.
  • Specific people and small organizations. The less written about them, the more the model fills in.
  • Recent events. Anything after the training data ends. The model may describe it anyway.
  • Your own systems. The model knows nothing about your internal tools, so it describes what such tools usually do.

A routine for checking

1. Decide what the answer is for. A first draft of a blog intro needs little checking. Anything that goes to a customer, a court, a regulator, a board or into production code needs every claim checked.

2. Open every source. Ask for sources, then open each one. Confirm it exists, that it says what the answer claims, and that it is current. A reference you did not open is not a reference.

3. Search for every quote. Look for the exact wording in the original. If you cannot find it, do not use it.

4. Recalculate every number that matters. Use a spreadsheet or code, not the model.

5. Ask again, differently. Regenerate the answer or rephrase the question. Facts the model actually knows tend to stay stable. Invented details tend to change between attempts. Disagreement between answers is a strong warning sign. A research paper released this spring uses this same idea to detect invented content automatically.

6. Ask what would make the answer wrong. A model asked “are you sure?” often just repeats itself or flips without reason. A model asked “what are the weakest points of this answer, and what evidence would contradict it?” gives you a list of things to check.

7. Give it the source instead of asking it to remember. The most effective fix is to put the relevant document into the prompt and require the model to answer only from that document. Then verify, in code, that its answer is supported.

Making the check automatic

When a model answers from a document you provide, you can require it to quote the exact sentences it relied on, and then confirm those sentences really appear in the document. If a quote is not in the source, the answer is rejected before anyone reads it.

import json
import openai

SYSTEM = (
    "Answer using only the document provided. "
    "Copy the exact sentences you relied on into the quotes list. "
    'If the document does not answer the question, reply {"answer": "not in document", "quotes": []}. '
    'Reply only with JSON: {"answer": "...", "quotes": ["...", "..."]}'
)

def normalize(text):
    return " ".join(text.split()).lower()

def answer_from_document(question, document):
    response = openai.ChatCompletion.create(
        model="gpt-3.5-turbo",
        temperature=0,
        messages=[
            {"role": "system", "content": SYSTEM},
            {"role": "user", "content": "DOCUMENT:\n" + document + "\n\nQUESTION:\n" + question},
        ],
    )
    result = json.loads(response["choices"][0]["message"]["content"])

    if result["answer"] == "not in document":
        return result

    if not result["quotes"]:
        raise ValueError("Answer has no supporting quotes. Rejected.")

    source = normalize(document)
    missing = [q for q in result["quotes"] if normalize(q) not in source]
    if missing:
        raise ValueError("Quotes not found in the document: " + "; ".join(missing))

    return result

This check proves that every quote exists in the source. It does not prove that the quotes support the conclusion, so a person still reads the answer next to its quotes. But it removes the worst failure, an answer built on text that was never there, and it does so before the answer reaches anyone.

The same idea scales up. When a system searches a library of documents first and passes the best matches to the model, you get answers grounded in your own material, with sources you can check. Without that grounding, the model answers from memory, and memory is where invented facts come from.

What to put in a team process

If answers from a language model feed decisions at your company, write down three rules:

  • A named person is responsible for every answer that is used, not the tool.
  • Every factual claim has a source that a person has opened.
  • The record shows what was checked. When a decision is questioned months later, “the AI said so” is not an answer. “We checked these three sources on this date” is.

A simple log is enough to start:

Claim Source opened Result Checked by Date
Vendor X supports SSO Vendor X admin guide, section 4.2 Confirmed J. Smith 2023-07-06
Regulation requires notice within 72 hours Text of the regulation, Article 33 Confirmed J. Smith 2023-07-06
Competitor raised prices 20% in 2022 No source found Removed J. Smith 2023-07-06

The third row is the one that matters. A claim with no source does not get softened or hedged. It gets removed.

Language models are excellent at drafting, explaining and summarizing. They are not, on their own, a source of facts. Use them for what they are good at, give them the material they need, and check the rest.