Every firm that has read one of the sanctions stories adds a line to the prompt: only cite real cases, don’t hallucinate, double-check every citation. It reads like a control. It isn’t one — and it fails hardest on the exact briefs where you were counting on it.
Many people are under the mistaken impression that an LLM is some sort of hypercomplex thinking machine. Really it’s an incredibly huge probability map of words, patterns, and phrases followed by other words, patterns, and phrases. Unless it’s attached to a lookup database of authoritative sources, nothing connects what it produces back to reality.
That’s the whole problem with the instruction. “Only cite real cases” asks the model to apply a filter that requires information it doesn’t have. The citation it can support and the citation it assembled come out of the same process and look identical from the inside — there is no internal flag on one that says invented. You have asked it to sort its output on a property it cannot observe.
Ask a person with no library to quote only from books they own, and the instruction does real work — they can walk to the shelf and see whether the book is there. Ask a model with no library, and you’ve added a sentence to a request, not a shelf.
Every model runs under a standing list of background directives. One of those directives is often referred to as "completion pressure" -- produce an answer rather than tell you it can’t. Hand it a place where a real authority should sit, and it fills the space instead of flagging it. That pressure is the engine behind every fabricated cite.
Adding “only cite real cases” doesn’t lift that pressure. It hands the model two orders that collide the moment real support runs out: reach the holding, and cite only real authority. On a well-supported argument, both are satisfiable and nothing happens. On the argument where the law is thin, one of them has to give — and the one that gives is the guardrail, because the guardrail is a preference and the conclusion is the assignment.
This is not carelessness. It is the model doing what you asked. A losing argument and a winning one look identical on the way in; only the availability of real support differs, and the model papers over that difference rather than tell you the support isn’t there. An instruction that gets overridden on precisely the briefs that needed it was never a safeguard.
These are the real ones — pulled from firm prompt libraries, CLE handouts, and the “AI policy” memo that went around after the first sanctions headline. Most briefs carry some combination of them.
>_Draft the argument section. Only cite real cases — do not hallucinate. Double-check every citation before you answer, give me a link for each one, and if you’re not sure about an authority, say so instead of guessing.
Nothing in that paragraph is wrong to want. Every clause fails for its own reason.
The model has no way to sort the citations it can support from the ones it assembled — both come out of the same place. You’ve asked it to filter on something it can’t see about its own answer.
This names the symptom and asks for the symptom to stop. Nothing in the request gives the model access to the authority it was missing, so the gap that produced the invention is still there.
The confidence is written the same way the citation is. A model that could tell you it wasn’t sure would already have known not to invent the case — and it will say “I’m confident” about the fabricated one.
The check is another pass over the same material. It re-reads its own answer, finds it consistent, and reports back that everything is in order — usually with a fresh summary of the case that doesn’t exist.
A link is one more claim in the answer. It can 404, it can open a different case, and it can open the right opinion that simply doesn’t contain the quoted words. Until somebody opens it and reads the page, it’s a promise, not proof.
The strongest of the bunch — a real constraint, and it cuts invented case names sharply. What it doesn’t constrain is the quoting: a real case from your memo, a pin cite that’s off, and language tightened into something the opinion never says.
The most trusted line in the whole prompt is the one that asks the model to verify its own work — “check each citation before you answer,” or the follow-up message every careful lawyer sends: are these cases real?
What comes back is another answer, not a lookup. The same map that produced the citation produces the confirmation, and it confirms. Push harder and it will describe the fabricated case for you — the posture, the holding, a plausible quote — because producing that description is the same kind of task as producing the cite was. Agreement between a model and itself is not evidence. It’s the same claim, twice.
This is why the sanctions record is full of attorneys who did ask. The question was answered confidently, in writing, by the thing that made it up.
Add the line, run twenty ordinary research questions, and the cites come back clean. That result is real — and it proves almost nothing. On a well-trodden question with abundant authority behind it, the model was going to cite real cases anyway; the instruction is riding along, not steering.
The brief that gets a firm sanctioned is never that brief. It’s the novel theory, the unusual jurisdiction, the element with no case squarely on point, the argument that has to reach. Those are the conditions where support runs out — which is exactly where the guardrail gives way. So the instruction earns your trust on the easy work and spends it on the hard work.
A test that always passes where failure is impossible is not measuring anything. The conditions that raise a brief’s hallucination rate are the same conditions that make the instruction fail.
Suppose the instruction did work half the time. You still couldn’t tell which half you were holding. A fabricated citation obeys every rule of form: a real-looking case name, a plausible reporter and volume, a page number in range, the right court and year in the parenthetical. Form is the part a probability map is superb at.
So the clause the model can satisfy is the one that makes the invention harder to spot, and the clause it can’t satisfy is the one you were relying on. The answer comes back looking more credible, not less.
And existence is only the loudest failure. A real case cited for a proposition it never reached, a real opinion with a quotation that isn’t in it, the right words attributed to the wrong page, a statute quoted in a version that was amended years before your filing date — none of those leave a mark on the page either. Reading the brief will not surface them. Reading the authority will.
Asking for a source link with every citation is a genuine step up, and a tool that actually searches before it writes is a bigger one. Invented case names get much rarer when something has to resolve.
What survives is the quoting. The link 404s. The link opens a different case with a similar name. The link opens the correct opinion, and the sentence in your brief inside the quotation marks isn’t anywhere in it — tightened, merged from two passages, or lifted from a dissent and cited as the holding. A URL sitting in a footnote reads as proof and functions as a promise. It becomes proof when someone opens it and reads the pin cite.
That is the step the prompt can never do for you, no matter how it’s worded. It has to happen against the authority itself, after the draft exists.
You wrote the instruction once, at the top. By page thirty the model’s dominant context is its own earlier output — and a case it invented on page four is now, as far as it’s concerned, part of the record. It cites it again, characterizes it further, and leans on it in the conclusion.
One sentence of guidance is competing against thirty pages of self-reference. In a multi-step or agentic workflow it’s worse: each step inherits the last one’s work rather than re-examining it, so the earliest invention is the one most likely to become load-bearing. The longer and more autonomous the generation, the further the instruction is from the moment it was needed.
Here is the part that makes this worth writing down: the instruction is not harmless. It lowers the fabrication rate a little, and it lowers how hard anyone checks a lot. The associate who knows the prompt says “only cite real cases” spot-checks four cites instead of all thirty-four. The partner hears that the firm’s AI policy addresses hallucinations and signs. The line in the prompt did most of its work on the humans.
A control that fails silently while raising everyone’s confidence is worse than no control at all. And when a citation is challenged, the conversation is about the document that was filed. What you asked the model for isn’t part of the record, and a prompt log has never been a verification record.
The fix isn’t a better sentence in the prompt. It’s a check that happens after the draft exists, against the authorities themselves. Verbatim reads a finished brief and reports, for every authority it cites, whether the cite is real and whether the quoted language actually appears at the pin cite — full cites, short forms, Id., supra, statutes and regulations at the version in effect on your filing date. Every verified cite carries a link straight into the source, so the page you were trusting is one click away.
It doesn’t ask the model whether the model was honest. It goes and looks — the same way, and produces the same report, every time you run it. Nothing depends on which words were in the prompt, which tool drafted the brief, or how many hands it passed through on the way to you.
So keep the instruction. It costs nothing and it helps at the margin. Just stop treating it as the thing standing between your firm and a show-cause order, and put a real check where that belief was.
Verbatim reads a finished brief and reports, for every authority it cites, whether the cite is real and whether the quoted language actually appears at the pin cite — so a fabrication surfaces on your screen, not in a show-cause order. Bring a brief and we’ll walk you through the report.