How to prevent AI hallucinations in business automation

You cannot make a language model stop hallucinating, but you can stop hallucinations from reaching customers, invoices and contracts. The approach that works in business automation is layered: give the model your own documents, constrain what it may output, check its claims in code, send anything uncertain to a person, and measure the error rate so you know whether the controls hold. This guide shows each layer, with a tested TypeScript verifier that decides whether an AI answer can be sent, needs review, or should become "I don't know".

Tested with: Node.js 24.19.0 (running TypeScript directly), TypeScript 6.0.3 (tsc --noEmit, strict) and @anthropic-ai/sdk 0.124.0. The verifier was run locally on responses shaped like the Messages API output; the API call was type-checked but not run against a live key.

Why hallucinations matter more in automation

In a chat window, a person reads the answer and may spot a mistake. In an automated workflow, nobody reads it: the model's output becomes an email, a refund, a ticket field or a line in a quote. The error is executed, not just displayed.

The legal position is also clear in at least one jurisdiction. In Moffatt v. Air Canada, 2024 BCCRT 149, British Columbia's Civil Resolution Tribunal held the airline responsible for its website chatbot's wrong statement that a bereavement fare could be claimed after travel, rejecting the argument that the chatbot was responsible for its own words. The tribunal found the airline "did not take reasonable care to ensure its chatbot was accurate". The amount was small; the principle is not. If your automation tells a customer something, your business said it.

Why models do this is now well described. OpenAI's September 2025 paper Why language models hallucinate argues that training and evaluation reward confident guessing over admitting uncertainty. A model that guesses a refund window scores better on many benchmarks than one that says "I don't know". Your system has to reverse that incentive: make abstaining a safe, normal outcome and make an unsupported claim fail.

The five controls

1. Ground the model in your own sources

Most business hallucinations are not exotic. The model is asked about your policy, price or stock level, which it cannot know, and fills the gap with something plausible. Retrieval fixes the input: fetch the relevant policy sections, product records or contract clauses and put them in the request. Our RAG pipeline tutorial covers chunking, embeddings and reranking, so here we focus on what happens after retrieval.

Grounding only helps if the instruction is strict. Anthropic's guidance on reducing hallucinations recommends three things that apply directly: explicitly allow the model to say it does not know, restrict it to the provided documents rather than general knowledge, and ask it to support claims with quotes, retracting any claim it cannot support.

Two practical rules from production systems:

• Retrieve from the system of record, not from a copy. A policy PDF exported last year is a fast route to a confidently wrong answer. Prices, stock and order status should come from your database at request time, through a tool call, not from a document index.
• Return nothing rather than something loosely related. If retrieval finds no passage above your relevance cut-off, skip the model call and answer "I don't know" or hand over to a person. Feeding a weak match invites the model to stretch it.

2. Constrain the output

Constraints remove whole classes of error. There are two main tools, and they do different jobs:

• Citations for free-text answers. With Claude's citations feature you attach documents with citations: { enabled: true }, and the response splits into text blocks, each carrying the exact passages it relies on. The documentation states that citations "are guaranteed to contain valid pointers to the provided documents", and the cited_text field does not count toward output tokens.
• Structured outputs for extraction and classification: invoice fields, ticket categories, yes or no decisions. A JSON schema means the output parses and every field has an allowed type or value. We covered schema design and its limits in getting valid JSON from LLMs. A schema guarantees shape, not truth: a perfectly valid "total": 1250.00 can still be wrong.

Note that you cannot use both on one request: the API returns a 400 if citations are enabled and output_config.format is set. If you need cited text and structured fields, make two calls, or extract with a schema and then verify each field against the source in code.

3. Verify in code

The model's own statement that it is confident is not evidence. Verification means a deterministic check outside the model. Useful checks, roughly from cheapest to most thorough:

• Coverage: what share of the answer is backed by a citation?
• Citation integrity: does each cited span exist in the source where it claims to be? Useful if you transform documents before sending them, or store and replay answers later.
• Figures: any amount, percentage, duration or date that appears in uncited text is the most expensive kind of hallucination in business, so treat it as a failure.
• Cross-checks against the system of record: an extracted invoice total must equal the sum of its lines; a quoted price must match the price table; a promised delivery date must be one the courier API actually returned.

Here is the verifier. It reads the content blocks returned by the Messages API and returns one of three decisions:

import type { ContentBlock } from '@anthropic-ai/sdk/resources/messages';

export interface SourceDoc { title: string; text: string }
export interface Verdict {
  decision: 'auto_send' | 'human_review' | 'no_answer';
  coverage: number;            // share of answer characters backed by a citation
  uncitedFigures: string[];    // amounts, dates, percentages and durations stated without a citation
  badCitations: string[];      // citations whose text is not found where they claim
  reasons: string[];
}

const ABSTAIN = 'NOT_IN_DOCUMENTS';
// Numbers that cost money when wrong: currency amounts, percentages, day/week/month counts, dates.
const FIGURE = /(?:[$£€]\s?\d[\d,]*(?:\.\d+)?|\d[\d,]*(?:\.\d+)?\s?(?:%|percent|days?|weeks?|months?|years?|hours?)|\b\d{1,2}\s(?:Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec)[a-z]*\b)/gi;

export function verifyGroundedAnswer(content: ContentBlock[], docs: SourceDoc[], minCoverage = 0.8): Verdict {
  const text = content.filter((b) => b.type === 'text');
  const answer = text.map((b) => b.text).join('').trim();
  const reasons: string[] = [];
  if (answer.includes(ABSTAIN)) return { decision: 'no_answer', coverage: 0, uncitedFigures: [], badCitations: [], reasons: ['model said the documents do not answer this'] };

  let cited = 0, total = 0;
  const uncitedFigures: string[] = [], badCitations: string[] = [];
  for (const block of text) {
    const len = block.text.trim().length;
    total += len;
    if (block.citations?.length) {
      cited += len;
      for (const c of block.citations) {
        if (c.type !== 'char_location') continue;           // plain-text documents return char_location
        const doc = docs[c.document_index];
        const span = doc?.text.slice(c.start_char_index, c.end_char_index);
        if (!doc || span !== c.cited_text) badCitations.push(`${c.document_title ?? '?'}: "${c.cited_text.slice(0, 40)}"`);
      }
    } else {
      uncitedFigures.push(...(block.text.match(FIGURE) ?? []));
    }
  }
  const coverage = total ? cited / total : 0;
  if (coverage < minCoverage) reasons.push(`only ${(coverage * 100).toFixed(0)}% of the answer is cited (minimum ${minCoverage * 100}%)`);
  if (uncitedFigures.length) reasons.push(`uncited figures: ${uncitedFigures.join(', ')}`);
  if (badCitations.length) reasons.push(`citations that do not match the source: ${badCitations.length}`);
  return { decision: reasons.length ? 'human_review' : 'auto_send', coverage, uncitedFigures, badCitations, reasons };
}

The call that produces those blocks attaches each policy as a plain-text document with citations on and gives the model a fixed abstain token:

import Anthropic from '@anthropic-ai/sdk';
import type { SourceDoc } from './verify.ts';

const client = new Anthropic(); // reads ANTHROPIC_API_KEY
const MODEL = process.env.CLAUDE_MODEL ?? 'claude-sonnet-4-6';

export async function answerFromPolicies(question: string, docs: SourceDoc[]) {
  return client.messages.create({
    model: MODEL,
    max_tokens: 1024,
    system:
      'You answer customer questions for a retailer using only the attached policy documents. ' +
      'Do not use general knowledge. If the documents do not contain the answer, reply with exactly NOT_IN_DOCUMENTS. ' +
      'Never state an amount, date or time limit that is not in the documents.',
    messages: [{
      role: 'user',
      content: [
        ...docs.map((d) => ({
          type: 'document' as const,
          source: { type: 'text' as const, media_type: 'text/plain' as const, data: d.text },
          title: d.title,
          citations: { enabled: true },
        })),
        { type: 'text' as const, text: question },
      ],
    }],
  });
}

And the wiring, which treats any stop reason other than end_turn (for example max_tokens or refusal) as a reason for review:

import { answerFromPolicies } from './grounded.ts';
import { verifyGroundedAnswer, type SourceDoc } from './verify.ts';

export async function handleQuestion(question: string, docs: SourceDoc[]) {
  const message = await answerFromPolicies(question, docs);
  if (message.stop_reason !== 'end_turn') return { decision: 'human_review' as const, reasons: [`stop_reason ${message.stop_reason}`] };
  const verdict = verifyGroundedAnswer(message.content, docs);
  // Log every verdict with the message id so you can measure rates later.
  console.log(JSON.stringify({ id: message.id, decision: verdict.decision, coverage: verdict.coverage.toFixed(2), reasons: verdict.reasons }));
  return verdict;
}

We ran the verifier against three responses built in the same shape as the API returns: a fully cited answer, an answer that invents a goodwill refund for sale items with a 14-day window and a $50 cap, and an abstention:

The invented policy is caught twice, by low coverage and by the uncited figures, and goes to a person instead of the customer.

4. Route by confidence and risk

"Confidence" in an automated system should mean evidence you can compute, not the model's self-assessment. Good signals are citation coverage, retrieval scores, agreement between two independent runs (Anthropic calls this best-of-N verification), and passing the cross-checks above. Combine them into a rule such as: send automatically only if coverage is at least 80%, no uncited figures exist and every cross-check passes; otherwise queue for review.

The threshold should depend on what a wrong answer costs:

For high-risk actions such as refunds, credit notes, price changes or contract terms, the model should prepare and a person should approve, every time. This is also the point where most teams need help with workflow design, since the review queue, permissions and audit trail are business software rather than prompting. Our AI business process automation service builds these approval steps into existing systems.

Regulation points the same way. In the EU, the AI Act's Article 50 transparency duties for systems that talk to people have applied since 2 August 2026, and high-risk systems carry human-oversight requirements under Article 14 (Regulation (EU) 2024/1689), although the high-risk deadlines have since been pushed back. Outside the EU, the Air Canada decision shows that ordinary consumer and negligence law already applies.

5. Measure

Without numbers you are guessing about your guesser. Measure at two points:

• Before release: an evaluation set of real questions, including ones the documents cannot answer and ones designed to tempt the model into inventing a figure. Score correct answers, correct abstentions and hallucinations separately; a system that never abstains is not accurate, it is lucky. Our practical guide to LLM evals covers building and automating that set.
• After release: log every verdict with the message ID, then track the review rate, the share of reviewed answers a person had to correct, and a weekly audit of a random sample of auto-sent answers. Every correction a reviewer makes is a new test case.

A rising review rate usually means your sources changed, not the model. A falling correction rate on reviewed answers is the evidence you need before you raise the auto-send threshold.

Worked example: a returns assistant

A mid-sized online shop wants an assistant that answers returns questions by email. The design that follows from the controls above:

1. Order status, delivery date and item type are fetched from the order system by ID. The model never guesses them.
2. The returns and warranty policies are attached as documents with citations enabled.
3. The verifier sends an answer automatically only if it is fully cited and every date it mentions matches the order record.
4. Anything involving money, such as a refund outside the window or a damaged-item claim, becomes a draft for a person to approve.
5. Questions the policies do not cover get a polite hand-over to the support team, not an improvised policy.

The model still does the useful part, reading a messy customer email and writing a clear reply, but it cannot make up a rule.

Common mistakes

• Asking the model whether it is sure. Self-reported confidence is poorly calibrated. Use computed signals.
• Treating valid JSON as correct JSON. Schemas fix format; cross-checks fix facts.
• Retrieving stale copies of live data. Prices and stock belong in tool calls, not in a vector index.
• No abstain path. If "I don't know" is not an allowed output, the model will produce something else.
• Review queues nobody owns. A human-review step without a named owner and a service level becomes a backlog, and someone will eventually switch it off.
• Trusting retrieved text as instructions. Documents and emails can contain text aimed at the model. Treat them as data; see our prompt injection guide.

When not to use an LLM at all

If the answer is a lookup, such as an order status, a price or an opening time, use a template filled from your database. If the decision is a fixed rule, such as "refund if within 30 days and unused", code it. Language models earn their place when the input is unstructured or the reply needs judgement about wording, not when a database query would do. A deterministic path never hallucinates.

Summary

Ground the model in current sources, constrain its output with citations or schemas, verify claims in code, route uncertain or high-risk outputs to people, and measure the results. None of these steps is complicated alone. Together they turn "the AI sometimes makes things up" into a known, monitored error rate with a person in the loop where it matters.

Want an AI workflow that answers customers or processes documents without inventing policy? Cloudsoft Zone builds AI business process automation with grounding, verification and human approval built in from the start.