How to reduce hallucination in a production LLM application?

WHY IT EXISTS Large language models generate the statistically likely next token, not a verified fact, so a model asked about something thin in its training data produces a fluent, confident answer that happens to be wrong. In production that becomes a wrong refund policy, an invented API parameter, or a fabricated citation. Hallucination mitigation exists because generating plausible text and generating true text are different goals that need separate engineering effort to converge.

THE MENTAL MODEL Treat hallucination the way you would treat any other reliability problem, with defense in depth rather than one fix. No single technique closes the gap between plausible and true, so production systems stack independent layers, each catching what the previous one misses, the same way a payments system never relies on client side validation alone.

HOW IT WORKS The first layer grounds the model in retrieved, trustworthy text through retrieval augmented generation, so the model is summarizing a document instead of recalling from memory. The second layer tunes generation itself, lowering temperature for factual tasks and fine-tuning on domain accurate examples so the model's default behavior favors your facts. The third layer verifies after generation, forcing structured output with citations, then programmatically checking that a quoted claim actually appears in the retrieved source before it reaches the user. A fourth layer closes the loop, logging low confidence or user flagged answers to improve the retrieval index and training data over time.

WHEN IT MATTERS This decides outcomes anywhere a wrong answer is costly: a support bot inventing a refund policy, a coding assistant inventing a function that does not exist, a medical or legal assistant stating something false with total confidence. The common interview red flag is naming only prompt engineering, phrases like just tell it to be accurate, or the opposite mistake, waving the problem away as fundamentally unsolvable instead of describing concrete mitigations.

ONE CONCRETE EXAMPLE A support bot is asked about refund windows. Instead of answering from memory, it retrieves the three most relevant help center articles, is instructed to answer only from that text and say it does not know otherwise, runs at a temperature of 0.2, and returns its answer with a link to the source article so the user can verify it directly.