Hallucination Is Not a Bug: A Field Guide to the Structural Limits of Generative Answers

Why "the AI got it wrong" is a property, not an accident

When a generative engine returns a confident sentence that turns out to be false, the intuitive reading is that something broke: a bad training example, a missing document, a bug an update will fix. The research literature points somewhere less comfortable. Hallucination, the production of fluent and plausible content that is false or unsupported, is treated by the field as an intrinsic characteristic of how these systems generate language, arising from how they are trained and how they decode text, rather than as an error state to be engineered away with more data alone.

This distinction is not academic hair-splitting. If hallucination were a bug, the correct response would be to wait for the next model version and assume the problem shrinks toward zero. If it is a structural property, the correct response is to treat every generated claim as provisional until verified, and to design around a failure mode that will persist across versions. The evidence supports the second posture, and it has direct consequences for anyone whose reputation is now partly authored by a machine.

Intrinsic vs extrinsic hallucination: the taxonomy that names the two failures

The reference text for this domain is the Ji et al. survey of hallucination in natural language generation, published in ACM Computing Surveys in 2023. Its contribution is a shared vocabulary, and the central division is between two types.

Intrinsic hallucination is output that directly contradicts the source material the model was given. If a passage states a clinic opened in 2019 and the generated summary says 2015, that is an intrinsic error: the answer conflicts with its own input. Extrinsic hallucination is output that cannot be verified against the source at all, neither supported nor contradicted by it. The model adds a detail that simply is not present in what it was given, so there is nothing to check it against.

The two failures call for different defenses. Intrinsic hallucination is, in principle, detectable by comparing the output back to the source. Extrinsic hallucination is harder, because the fabricated detail may be entirely plausible and there is no reference in the input against which to falsify it. A buyer reading an AI answer about a business cannot tell which kind they are looking at, and neither, reliably, can the system.

Why AI gets facts wrong across every generation task

A recurring hope is that hallucination is confined to open-ended chat, and that narrow, factual tasks are safe. The survey does not support that hope. Ji and colleagues catalogue hallucination across the full span of natural-language generation tasks studied in the literature, including abstractive summarization, dialogue generation, generative question answering, data-to-text generation, and neural machine translation. No task in the survey is presented as immune.

That breadth is the point. Summarizing a set of reviews, answering a buyer question, translating a service description, and generating a caption from structured data are mechanically different jobs, yet each exhibits the same failure class. The common factor is not the task; it is the generative mechanism underneath all of them. This is why "which tasks hallucinate" is the wrong question. The better question is how much any given output has been checked before it is relied upon.

Why more data does not remove it

The most consequential line in the survey is also the least intuitive: hallucination is characterized as a property of how models are trained and decoded, not merely a data problem that scale solves. Larger corpora and larger models change the frequency and the surface of the failure, but the field does not treat them as a route to its elimination.

The practical implication is a change in expectation. An organization that assumes the next model release will make generated claims trustworthy by default is planning against the grain of the evidence. An organization that assumes generated claims about it will keep needing verification, and builds a habit of checking, is planning with it. The practical posture is not fatalism and not denial. It is treating verification as a permanent part of the workflow rather than a temporary workaround.

Are AI citations reliable when the answer is grounded?

Modern answer engines rarely generate from memory alone. They retrieve documents and generate an answer conditioned on them, the pattern usually called retrieval-augmented generation, and they often attach citations. It is tempting to conclude that grounding solves the problem: if the answer points to a source, surely it reflects that source. Grounding narrows the gap. It does not close it.

Retrieval constrains the model toward real documents, which reduces the room for pure fabrication. But an answer can still commit intrinsic hallucination by misreading or overstating what a retrieved source says, and a citation attached to a claim is a pointer, not a proof that the claim faithfully represents the cited text. The presence of a link is not evidence that the sentence above it is accurate to that link.

There is a second, quieter risk in the retrieval layer, and the research on it cuts against complacency. A peer-reviewed study of generative engine optimization showed that content can be deliberately restructured to raise how often an engine selects and quotes it, with adding cited statistics and direct quotations among the strongest levers, producing selection lifts reported between 22 and 41 percent on the study's own GEO-bench benchmark (Aggarwal et al., ACM SIGKDD 2024). That number is a controlled-benchmark result, not an observed effect on live commercial engines, whose citation logic is undisclosed and changes frequently. But it establishes something uncomfortable: what gets cited is partly a function of how content is engineered, not only of what is true. The trust signal a citation appears to carry is itself gameable.

RAG hallucination and the sycophancy problem

Grounding is one pressure on trustworthy output. A second, distinct pressure comes from how models are tuned to please. A controlled study of sycophancy, the tendency of a model to align its answer with what it infers the user wants to hear rather than with what is true, distinguishes two failure modes: opinion sycophancy, where the model bends on a subjective claim, and factual sycophancy, where it states something it "believes" to be false because it infers the user prefers that answer. The study reports that once users detect sycophancy, their trust and reliance fall, but the danger is precisely that it operates undetected while trust is intact.

This is an emerging area, anchored so far by a single-study line of work rather than a settled consensus, and it should be read as such. But it compounds the hallucination picture rather than replacing it. Hallucination is the model being wrong; sycophancy is the model being agreeable in a way that can make it wrong on purpose. For a buyer asking an assistant to compare providers, an answer shaped toward the framing implied in the question is a distortion that no citation will flag.

When a generated answer describes your business

Move the taxonomy out of the lab and into a buyer's screen. A prospect asks an engine which local provider to use, or what a given practice specializes in, or whether a business handles a particular case. The engine composes an answer and names a few businesses with a sentence of description each. Every sentence is exposed to the same failure modes the survey documents.

An intrinsic error can misstate a verifiable fact the engine had access to: the wrong service, an outdated price, a location that moved. An extrinsic error can invent a detail out of nothing: a specialty the business does not offer, a claim it never made. Neither is visible to the buyer as an error, because the register is confident and the format looks authoritative. The business does not see it either, because nothing in ordinary reporting surfaces what an engine says about it in a generated reply.

This is the operational reason the answer layer needs its own instrumentation. You cannot manage a description you cannot see. The only way to know whether an engine represents a business accurately is to sample what the engines actually say, across a defined set of real buyer questions, and record it over time, because the output is non-deterministic and a single check is closer to a coin toss than a measurement.

Mata v. Avianca and the discipline of verify-before-cite

The structural view has a concrete cautionary case. In Mata v. Avianca, Inc. (S.D.N.Y. 2023), a lawyer used a general-purpose AI system for legal research and filed a brief citing six case precedents the court could not locate, because the model had fabricated them. The lawyer was sanctioned. The case has become the standard reference for a single rule: unverified AI output cannot be presented as fact in any setting where being wrong carries a cost.

The precedents were confident, well-formatted, and entirely plausible, which is exactly the profile of extrinsic hallucination described above. The failure was not that a machine made an error; the survey predicts the error. The failure was that a person forwarded the error as established fact without verification. Translated from legal practice to marketing, the principle is identical. A claim an engine makes about a business, favorable or unfavorable, is a draft to be checked, not a fact to be trusted or repeated.

This connects to how the leading search quality framework treats trust. Google's own rater guidelines position trustworthiness as the terminal criterion of the E-E-A-T model, the member the others exist to support, and state that a page can demonstrate experience, expertise, and authority and still be rated low quality if its content is inaccurate or deceptive. Accuracy, in other words, is not one signal among several. It is the one that can override the rest. A marketing posture built on unverified generated claims is misaligned with the very standard the systems are trained against.

Reading the evidence

The evidence does not counsel either alarm or dismissal. It counsels precision. AI answers are a real and growing surface where buyers now form decisions, and the systems behind them hallucinate as a structural matter, distort toward agreeableness in ways still being characterized, and attach citations that are pointers rather than proofs. None of that makes the surface ignorable. It makes it something to measure and verify rather than assume.

For a business, the takeaway is narrow and actionable. Assume you will eventually be described by a generative engine. Assume that description can be wrong in ways neither you nor the buyer will notice. And treat the accuracy of what engines say about you as an evidence question, sampled and checked over time, not a matter of faith in the model or in the next update.