How to Reduce LLM Hallucinations in Production
Language models state wrong things fluently, and in production that becomes a support or legal problem. This guide explains why models hallucinate, then asks a small model twelve questions about an invented shop's policies under four set-ups, from no document to a document with rules and a quote check in code.
GloryTecks Team
Training & Placement Faculty
Last updated: 3 October 2026
A language model is built to produce plausible text. When it knows the answer, plausible and true coincide. When it does not, it usually produces something that sounds just as sure. In a chatbot that answers questions about your products or policies, that is a customer told the wrong refund window, in your company's voice.
Hallucinations cannot be removed entirely, but in production they can be made much rarer and much easier to catch. This guide explains where they come from, then measures four set-ups on the same questions.
Why models make things up
A survey of the research defines the problem as generating plausible yet nonfactual content, and sorts its causes across data, training and generation (Huang et al.: hallucination survey). A 2025 paper from OpenAI researchers gives a sharper account of why it persists. Models are trained and evaluated in ways that reward guessing: on most benchmarks, a confident guess can score and "I don't know" cannot. Like students facing a hard exam, models learn to guess (Kalai et al.: Why Language Models Hallucinate).
For an application, that suggests three levers. Give the model the facts, so it has no need to guess. Make "I don't know" an acceptable answer. And check answers in code, so that a guess cannot pass as a fact.
Four set-ups, twelve questions
The test uses a handbook for an invented furniture shop: returns, refunds, delivery, warranty and so on. Because the shop does not exist, no model can know its policies from training. Eight questions are answered in the handbook and four are not.
The four set-ups add one lever at a time:
| Set-up | What the model gets |
|---|---|
| A | The question only |
| B | The handbook in the prompt |
| C | The handbook, a rule to answer only from it or say NOT IN HANDBOOK, and a JSON reply with a supporting quote |
| D | The same reply as C, but code checks that the quote really appears in the handbook before the answer is used |
import json
import re
import urllib.request
URL = "http://localhost:11434/api/chat"
MODEL = "llama3.2"
OPTIONS = {"temperature": 0, "seed": 42, "num_predict": 150}
# The policy of an invented shop: no model can know these facts from training
HANDBOOK = """Larkfield Furniture customer handbook
Returns: items can be returned within 21 days of delivery, unassembled and in original packaging.
Refunds: paid to the original payment method within 7 working days of the return arriving at our warehouse.
Delivery: free on orders over 10,000 rupees; otherwise delivery costs 499 rupees.
Assembly: our assembly service costs 999 rupees per item and is available in Pune and Hyderabad only.
Warranty: chairs 3 years, desks 5 years, lamps 1 year.
Damage: report damaged items within 48 hours of delivery, with photos.
Support hours: 9:00 to 18:00, Monday to Saturday.
Gift cards: valid for 12 months; they cannot be exchanged for cash."""
# (question, accepted answers); None marks a question the handbook does not answer
QUESTIONS = [
("How many days do I have to return an item?", ["21"]),
("How long does a refund take once my return arrives?", ["7 working days", "seven working days"]),
("How much is delivery on a 6,000 rupee order?", ["499"]),
("How much does your assembly service cost?", ["999"]),
("How long is the warranty on an office chair?", ["3 years", "three years"]),
("Until what time is support open on Saturday?", ["18:00", "6 pm", "6:00 pm", "6pm"]),
("How soon must I report a damaged item?", ["48 hours"]),
("How long is a gift card valid?", ["12 months", "twelve months", "one year", "1 year"]),
("Do you deliver to Kolkata?", None),
("What is the warranty on a monitor arm?", None),
("Can I pay in monthly instalments?", None),
("Do you match competitors' prices?", None),
]
ABSTAIN = re.compile(r"not in (the )?handbook|don't have|do not have|don't know|not sure|no information|"
r"not (mentioned|specified|stated)|doesn't (say|mention|specify)|does not (say|mention|specify)|"
r"unable to|can't (find|provide)|cannot (find|provide)|not aware|contact (our|the|us)", re.I)
RULES = ("Answer only from the handbook below. If the handbook does not answer the question, the answer must be "
"exactly NOT IN HANDBOOK. In quote, copy the exact handbook words that support the answer, or leave it empty.")
SCHEMA = {"type": "object", "properties": {"quote": {"type": "string"}, "answer": {"type": "string"}},
"required": ["quote", "answer"]}
def ask(system, question, schema=None):
body = {"model": MODEL, "stream": False, "options": OPTIONS,
"messages": [{"role": "system", "content": system}, {"role": "user", "content": question}]}
if schema:
body["format"] = schema
req = urllib.request.Request(URL, json.dumps(body).encode(), {"Content-Type": "application/json"})
with urllib.request.urlopen(req, timeout=900) as r:
return json.load(r)["message"]["content"]
def grade(answer, accepted):
text = answer.lower().replace(",", "")
if accepted is not None and any(re.search(rf"(?<![\d:]){re.escape(a)}(?![\d:])", text) for a in accepted):
return "right"
if not answer.strip():
return "blank"
return "abstained" if ABSTAIN.search(answer) else ("wrong" if accepted is not None else "invented")
def supported(quote):
squash = lambda s: " ".join(s.lower().split())
return len(quote.strip()) >= 8 and squash(quote) in squash(HANDBOOK)
results = {k: [] for k in "ABCD"}
examples = {}
for question, accepted in QUESTIONS:
a = ask("You are the customer assistant of Larkfield Furniture.", question)
b = ask("You are the customer assistant of Larkfield Furniture.\n\n" + HANDBOOK, question)
c = json.loads(ask("You are the customer assistant of Larkfield Furniture. " + RULES + "\n\n" + HANDBOOK, question, SCHEMA))
# D: the same reply as C, but the application checks the quote before trusting the answer
d = c["answer"] if supported(c["quote"]) else "NOT IN HANDBOOK"
for key, answer in zip("ABCD", (a, b, c["answer"], d)):
results[key].append(grade(answer, accepted))
examples[question] = (a, b, c, d)
names = {"A": "no handbook", "B": "handbook in the prompt", "C": "handbook, rules, quote", "D": "C plus a quote check"}
print(f"{'':24} {'8 answerable questions':^32} {'4 the handbook does not answer':^26}")
print(f"{'':24} {'right':>6} {'wrong':>6} {'abstain':>8} {'blank':>6} {'abstain':>8} {'blank':>6} {'invented':>9}")
for key, label in names.items():
ans, una = results[key][:8], results[key][8:]
print(f"{label:<24} {ans.count('right'):>6} {ans.count('wrong'):>6} {ans.count('abstained'):>8} {ans.count('blank'):>6} "
f"{una.count('abstained'):>8} {una.count('blank'):>6} {una.count('invented'):>9}")
def one_line(s, n=150):
return " ".join(s.split())[:n] or "(blank)"
print("\nno handbook:", QUESTIONS[0][0])
print(" " + one_line(examples[QUESTIONS[0][0]][0]))
print("\nhandbook in the prompt, the questions it does not answer:")
for question, _ in QUESTIONS[8:]:
print(f" {question}\n {one_line(examples[question][1])}")
print("\nrules and a quote, the same questions:")
for question, _ in QUESTIONS[8:]:
c = examples[question][2]
print(f" {question} answer: {one_line(c['answer'], 60)} quote: {one_line(c['quote'], 40)}") 8 answerable questions 4 the handbook does not answer
right wrong abstain blank abstain blank invented
no handbook 1 4 3 0 0 0 4
handbook in the prompt 8 0 0 0 1 0 3
handbook, rules, quote 8 0 0 0 0 4 0
C plus a quote check 8 0 0 0 4 0 0
no handbook: How many days do I have to return an item?
At Larkfield Furniture, we want you to be completely satisfied with your purchase. We offer a 30-day return policy. If you need to return or exchange
handbook in the prompt, the questions it does not answer:
Do you deliver to Kolkata?
Unfortunately, we do not currently offer delivery services to Kolkata. Our standard delivery areas are limited to major cities in India such as Pune a
What is the warranty on a monitor arm?
I don't have information on the warranty of a specific product like a monitor arm from Larkfield Furniture. However, I can tell you that we do offer a
Can I pay in monthly instalments?
I'm afraid we don't currently offer monthly installment options for our products. However, we do have a financing option available through our partner
Do you match competitors' prices?
At Larkfield Furniture, we strive to offer competitive pricing while maintaining the quality and value that our customers have come to expect from us.
rules and a quote, the same questions:
Do you deliver to Kolkata? answer: (blank) quote: (blank)
What is the warranty on a monitor arm? answer: (blank) quote: (blank)
Can I pay in monthly instalments? answer: (blank) quote: (blank)
Do you match competitors' prices? answer: (blank) quote: NOT IN HANDBOOKWhat the output shows
- Without the handbook, the model answered anyway. It stated a 30-day return policy for a shop whose policy is 21 days, and gave confident answers to every question the handbook does not cover. Only three times did it admit it could not know.
- The handbook fixed the answerable questions. With the document in the prompt, all eight were right.
- The document did not stop invention. Asked things the handbook does not cover, such as delivery to Kolkata or a monitor arm's warranty, the model still filled the gaps in three of four cases. It said delivery was limited to major cities such as Pune, and offered a financing option through a partner. The handbook mentions neither. Grounding helps with what is in the document. It does nothing about what is not.
- Rules and a quote field stopped the invention, but not neatly. The model kept all eight right answers. For the four uncovered questions it did not invent, but it left the answer blank instead of writing NOT IN HANDBOOK as told. Once it even put that phrase in the quote field.
- The check in code turned those blanks into a clear, correct reply. Each answer was accepted only with a quote found word for word in the handbook. All eight right answers had such quotes and survived. The four unsupported ones became "not in the handbook".
A blank answer would have reached a customer as an empty message, or been filled by a fallback. The last step is the one that made the behaviour dependable, and it is ordinary code.
What reduces hallucinations in production
Ground answers in your own content. Retrieve the relevant documents and put them in the prompt. This is the job of retrieval-augmented generation, and its quality sets a ceiling on everything else. See RAG architecture explained.
Give the model a way out. Anthropic's guidance starts with explicitly allowing the model to say it does not know, and restricting it to the provided documents (Anthropic: reduce hallucinations).
Ask for quotes, then check them. The same guidance suggests extracting word-for-word quotes, verifying each claim against a supporting quote, and retracting claims that have none. Doing the check in code, as set-up D did, makes it deterministic.
Use structured output for anything code will read. A field for the answer and a field for the evidence are easier to validate than a paragraph.
Compare several runs on high-stakes answers. Inconsistent answers to the same question are a warning sign, and Anthropic lists comparing outputs as a check.
Decide what happens on "not found". Hand over to a person, link to the full policy, or say plainly that the information is not available. Never fall back to letting the model guess.
Measure it
Hallucination rates are only known if they are measured. Keep a set of questions with known answers, including questions your documents do not answer, and check every change of model, prompt or retrieval against it. Count invented answers separately from wrong ones. They have different causes and different fixes. See evaluating LLM outputs. In production, sample real answers and their sources for review, and give users a way to report errors.
Deliberate manipulation, where someone tries to make the model say something false, is a separate problem with separate defences. See guardrails for LLMs.
Common mistakes
- Assuming retrieval solves it. Questions your documents do not answer still get answers.
- No permission to abstain, so the model always produces something.
- Trusting citations without checking them. A model can quote text that is not in the source.
- Measuring accuracy only on answerable questions.
- Letting a blank or "not found" response fall through to a generic model answer.
- Fine-tuning to add facts, when the facts change and retrieval would keep them current. See fine-tuning vs RAG.
What is an LLM hallucination?
Does RAG eliminate hallucinations?
How do I detect hallucinations automatically?
Do bigger models hallucinate less?
Hallucination reduction is part of the Responsible and Safe AI module of the Generative AI course.
Sources
- Why Language Models Hallucinate (arXiv) — Kalai et al.
- A Survey on Hallucination in Large Language Models (arXiv) — Huang et al.
- Reduce hallucinations (Claude Platform Docs) — Anthropic
GloryTecks Team
Training & Placement Faculty
The GloryTecks faculty team mentors learners across Data Science, AI and Data Engineering, with a focus on real-time projects and placement readiness in Hyderabad.
Want to master Generative AI?
Join GloryTecks’ expert-led Generative AI training in Hyderabad with real-time projects and 100% placement support.
Related articles
Evaluating LLM Outputs: Metrics, Tools and Best Practices
Every change to a prompt or model needs evidence that it helped. This guide explains the main ways to evaluate LLM outputs, then checks a small local LLM judge against expert human verdicts from MT-Bench and finds it agreeing often, flipping often when answers swap places, and trustworthy only when consistent.
What Is Generative AI? A Complete Beginner's Guide
What generative AI is, in plain language: how language models produce text one token at a time (shown with a tiny runnable model), how chat models are trained, how image models differ, what the technology is reliable for, and why it confabulates.
ChatGPT Complete Guide: Features, Prompts and Use Cases
ChatGPT went from a research preview to one of the most used software products in the world in under four years. This guide maps its main features and when they arrived, shows how to write prompts that work, sorts good uses from risky ones, and explains the limits every user should check for.