The citations that don't exist

It is one of the most common requests anyone makes of a language model, and one of the most dangerous, because the output is shaped exactly like a citation. A plausible title. A surname and "et al." A journal that exists. A year. A DOI in the right format, with the right prefix, the right slash, the right length.

Everything that normally signals "somebody checked this" is present, and none of it was produced by checking anything. The reason people keep getting caught — including lawyers, in court — is that a fabricated reference does not look fabricated. It looks like a reference.

The obvious method is to search for the title and call it fabricated when you cannot find it. That method has a hole large enough to invalidate the whole result, and the hole is not evenly distributed.

Bibliographic databases do not index everything. African journals, theses, institutional reports and conference volumes are covered patchily. So a real paper on a Ghanaian topic can fail a title search — which means a naive method would manufacture exactly the asymmetry this page is looking for, and I would have no way to tell the difference.

So the primary measurement is not a search. It is the DOI.

This page was written expecting to count fabrications. The first run made that the second question, because on the narrow topics most of the models did not produce references at all. They declined — and said why.

Now the original question, asked of what came back. Every claimed reference resolved against the registry and four indexes.

The topics come in matched pairs. Same field, same kind of question, different amount of published work behind it — "statin therapy for the primary prevention of cardiovascular disease" against "community case management of malaria in northern Ghana", and five more like it. The pairing is the control: comparing medicine against Ghanaian linguistics would confound the field with the thinness, so each comparison stays inside one field.

This is the objection that decides whether Section 4 means anything, and it nearly sank the measurement. If the indexes I search are worse at Ghanaian research than at American research, then a higher "fabricated" rate on Ghanaian topics is a fact about my tooling.

It was, the first time I ran it. The first version searched two indexes, swallowed every error, and returned the best of whatever came back. OpenAlex — which has since moved to a metered plan — started returning rate-limit errors partway through, silently turning a two-index search into a one-index search. Papers that certainly exist came back as missing, and the miss rate looked like a finding. It was a bug.

The fix has two parts. Four unmetered indexes instead of one metered one, and a hard rule: a "not found" is only recorded when every index actually answered. If one errors, the citation is marked unresolved and scored as nothing at all.

Then the control. Take papers that are known to be real — sampled from PubMed and arXiv, neither of which is one of the resolvers — and run their titles through the same search. Whatever it fails to find is the false-fabrication rate.

Everything above was answered from the model's weights, with no tools. That is how a model behaves in a plain chat window, which is how most people ask. But the same question can be put to the same models with web search switched on.

If grounding collapses the invented rate, then the honest headline is not about which model you chose. It is about which mode you were in — and that is a far more useful thing for a reader to know.

A DOI that fails to resolve is the harmless failure: you click it, you get an error, you know. The one that costs people their credibility is the DOI that works and belongs to a different paper. You click, a real paper loads, and everything looks confirmed — unless you read the title that came up and compare it with the title you were given.

Resolve every DOI before you use it. One request to doi.org/api/handles/<doi> answers the only question that has an unarguable answer. It is free and takes milliseconds.

Then compare the titles. A resolving DOI is necessary, not sufficient — Section 7 is the reason. Check that the paper that loads is the paper you were told about.

Trust the answer less as the topic gets narrower. —

Prefer the grounded mode when sources matter. —

Treat "NONE" as good behaviour, not a failure. A model that declines to produce an identifier it does not have is telling you something true. The failure mode is confidence, not silence.

Ten questions on the method and the findings. The explanations are worth reading even when you get it right.