Build log/AI

How to design AI features for when the answer is wrong

Every AI feature will sometimes be wrong. Design the screen for that day, with sources shown, outputs as editable drafts, facts checked in code and actions that can be undone.

The short answer

  • A hallucination is when a language model produces a fluent, confident statement that is false or not supported by the information it was given.
  • Treat every AI output as a draft that a person can trace to its source, edit and reject, until testing shows a step is safe to automate.
  • Check numbers, prices, dates and permitted actions in code, so a model’s output is never the only thing between an error and a customer.

Designing for wrong answers means building an AI feature so that when the model is mistaken, a person notices before it matters and can fix it in seconds. In practice that means showing where each answer came from, presenting outputs as editable drafts, checking key facts in code and making actions reversible. A hallucination is a fluent, confident output from a language model that is false or not supported by the information it was given.

Why AI answers are sometimes wrong

A language model writes by predicting likely next words, based on patterns learned in training and on whatever text it receives in the prompt. It doesn’t look facts up unless your software gives it a way to, and it has no reliable sense of when it doesn’t know. When the information it needs is missing, it tends to produce something that sounds right.

That is why a model asked to read a supplier invoice may return a plausible VAT number when the invoice doesn’t show one. Nothing in the output marks the invented value as different from the real ones beside it. Better models and better prompts reduce how often this happens. They don’t reduce it to zero, so the screen has to be designed for the cases that get through.

Show where every answer came from

The most useful single feature is a visible source. If an answer is built from documents or records, show the passage or field it used next to the claim, and let the person click through to it. This is one of the main practical benefits of retrieval, described in RAG explained.

Sources change how people review. A reviewer who can see that a £4,200 contract value came from page 3 of the uploaded tender can check it in two seconds. A reviewer shown only the number has to trust it or go hunting. Where an answer has no source, say so on the screen in words such as “not found in your documents” or “general answer, not from your records”.

For extraction features, a blank is better than a guess. If a field can’t be found in the source, leave it empty and highlight it. Code can enforce this by checking that each extracted value really appears in the source text before the field is filled.

Review states and confidence

Every AI output should carry a visible state, so nobody mistakes an unchecked draft for something a person has approved. A simple set of states covers most features.

StateWhat it meansWhat the screen does
DraftGenerated by AI, not yet looked atLabelled clearly as AI-generated; can’t be sent or posted
Needs attentionA check in code failed or a field is missingHighlights the specific problem and where to fix it
ReviewedA named person has checked and accepted itRecords who and when, and enables the next action
Sent or appliedIt has had an effect outside the systemShows the history and offers undo where possible

Be careful with confidence scores. A model asked how sure it is will give a number, but that number is generated text like any other, and it is often a poor guide to accuracy. Better signals come from your own code. Did the value pass validation? Does it appear in the source? Do two independent methods agree? Do the line items add up to the total? Use those checks to decide what lands in “needs attention”.

Editable drafts and undo

The simplest protection is a person between the model and the outcome, with a screen that makes their job quick. An AI output should open as an editable draft, with editing as the obvious action and accept-and-send as a deliberate second step.

CurioKids works this way for lesson content. Content is generated for each child from their CEFR level, age and interests, and the teacher can change it. The model does the groundwork, and the teacher can change whatever doesn’t suit the child. The thinking behind that feature is in how a lesson is shaped for each child.

Undo matters most where AI acts directly. Keep previous versions of anything the AI changed, so one click restores the old text. Prefer soft deletes, which hide a record, to hard deletes, which destroy it. Where an action can’t be reversed, such as an email that has already been delivered, that is exactly where a human approval step belongs.

Guardrails in code

Guardrails are checks in your own software that the model can’t argue its way past. They sit between the model’s output and anything that happens next.

  • Validate structure. Require output in a defined format and reject anything that doesn’t match it.
  • Recalculate money. Compute totals, VAT, CIS deductions and discounts in code from the line items, and never accept them from the model.
  • Check against your records. A product code, customer or price the model mentions must exist in your database.
  • Allow-list actions. The model can request only actions you have defined, and each request is checked against the user’s permissions.
  • Set limits. Cap refund amounts, discount levels and message counts, so the worst case has a known size.
  • Filter what is shown. Screen outputs for content that should never appear, especially in products used by children.

Feedback loops that improve the feature

Thumbs up and thumbs down buttons are easy to add and easy to ignore. The richer signal is what people do with the output. Record whether each draft was accepted unchanged, edited or thrown away, and keep the edited version. Comparing the draft with what the person actually sent shows exactly where the model falls short, without asking anyone to fill in a form.

Ask for a reason only when someone rejects a draft, and make it one tap from a short list, such as “wrong facts”, “wrong tone” or “missing information”. Feed those cases into your evaluation set, so a fix for one failure is tested against everything that already worked. The process is set out in how to test an AI feature.

How the screen should behave

Pulling this together, an AI feature that handles wrong answers well follows a short list of rules.

  1. Label AI-generated content as AI-generated, in words.
  2. Show the source next to each fact, or say plainly that there isn’t one.
  3. Leave fields blank and highlighted when the source doesn’t contain them.
  4. Keep the original input visible beside the output.
  5. Make editing the obvious next step.
  6. Never send, publish or charge without a deliberate action by a person.
  7. Keep the history, and offer undo wherever the action allows it.
Adding AI to a product people rely on? We design the review screen at the same time as the prompt, because it decides whether the feature can be trusted. See how we build AI features.

Questions people ask

Can AI hallucinations be prevented completely?

No current method removes them entirely. Better models, retrieval from your own documents and careful prompts make them less frequent. Design the feature so a wrong answer is visible, easy to correct and unable to cause harm on its own.

Should we show a confidence score to users?

Only if it comes from checks you can trust. A model’s own estimate of its confidence is often unreliable. Signals from code, such as whether a value was found in the source or whether totals reconcile, make better warnings.

When is it safe to let AI act without review?

When the action is low risk, easy to undo, and testing on real cases shows the feature is accurate enough for that job. Sorting emails into folders may qualify. Sending a quote or issuing a refund usually doesn’t.

Who is responsible when an AI feature gets something wrong?

In general, the business using the output remains responsible for what it tells customers or decides about them. That is a strong reason to keep a person in the loop for anything with legal, financial or safety consequences. Take legal advice where the stakes are high.

Written by Flavoureak

We build and run Estimark, CurioKids and CalcStack, and we build web and mobile software for other businesses. Start a project.