Build log/AI

How to test an AI feature before your customers see it

Build a test set from real cases, agree what a pass looks like, score every change against it and keep watching once it is live. Most of the work is the test set.

The short answer

  • An evaluation set is a fixed collection of real inputs, each with the result you expect, that you run an AI feature against before every release.
  • Use automated checks for anything a rule can verify, such as format, required fields and numbers, and keep human review for tone, judgement and subtle errors.
  • Rerun the whole evaluation set whenever the prompt, the model or the retrieved data changes, because a fix in one case often breaks another.

You test an AI feature by running it against a fixed set of real cases, each with an agreed definition of a correct result, and counting how many it gets right before anything reaches a customer. That fixed set is called an evaluation set, or eval set. It is a collection of example inputs, each paired with an expected output or a written rule for judging the output, and it is rerun every time the prompt or the model changes.

Why AI features need a different kind of testing

Ordinary software is deterministic. Give a VAT calculation the same invoice twice and it returns the same figure twice, so one test proves the rule works. A language model can return a different answer to the same input, and a change that improves one answer can quietly make another worse. A good demo on five hand-picked examples proves very little.

The answer is to test with volume and to test the same way every time. The useful question is how many of 150 real cases the feature gets right, and which ones it gets wrong. That number, and the list of failures behind it, becomes the thing the team protects from one release to the next.

How to build an evaluation set from real cases

The best test cases are the ones the feature will actually meet. Examples written by the team tend to be tidy, and real inputs are not. Emails arrive with signatures and forwarded chains. Documents arrive as phone photos taken at an angle. Customers ask two questions in one message and leave out the order number.

  1. Collect real inputs. Pull past support tickets, documents, enquiries or records that the feature would have handled. Remove or replace any personal data the test doesn’t need.
  2. Write down the expected result for each one. For extraction, that is the correct value of every field. For drafting, it is a short list of what a good answer must include and must not include.
  3. Cover the awkward cases on purpose. Include blank fields, contradictory information, very long inputs, other languages and requests the feature should decline.
  4. Tag each case. Label it by type, such as “handwritten”, “two questions” or “out of scope”, so you can see where failures cluster.
  5. Version it. Store the set alongside the code, and add every real failure found later as a new case.

Fifty well-chosen cases are enough to start. A few hundred give a clearer picture of rare failures. The set should keep growing for as long as the feature is live, because production keeps finding inputs nobody thought of.

Deciding what counts as a pass

Pass criteria have to be written before anyone looks at the results. Otherwise the team will talk itself into accepting whatever came out. Write them per case and per feature, and make each one something two reviewers would agree on.

Take a feature like the lesson content in CurioKids, which is generated for each child from their CEFR level, age and interests, and which the teacher can change. A pass for a case like that can be broken into checks. The vocabulary suits the stated level. The topic reflects at least one of the child’s interests. Nothing in it is unsuitable for a child. The activities fit the length of the lesson. Each check is met or not met, which is far easier to agree on than a score out of ten.

Then set a release threshold for the whole set. Say the set has 150 cases and the current version passes 138 of them, which is 92%. A new version ships only if it passes at least as many, and only if every case that has gone from pass to fail has been read and accepted by a named person.

Automated checks and human review

Some checks a computer can make instantly and without disagreement. Others need a person. The aim is to automate everything a rule can decide, so that human time goes on the judgements that really need it.

CheckWho or what does itExample
Format and structureCodeThe output is valid JSON with every required field present
Exact valuesCodeThe extracted invoice total matches the expected £1,240.00
Numbers within toleranceCodeA measured quantity is within 1% of the expected figure
Must include or must avoidCodeA reply mentions the refund window and never promises a delivery date
Meaning and coverageA second model, spot-checked by peopleA summary covers the three key points of the source document
Tone, judgement and safetyPeopleA reply to an upset customer is accurate, calm and appropriate

Using a second model to grade outputs, often called model-graded evaluation, is useful for checking meaning at scale. It makes mistakes of its own, so check a sample of its grades by hand. Don’t let it be the only judge of anything that affects money, safety or a customer’s rights.

For human review, show reviewers the input, the output and the written criteria side by side, and ask for pass or fail against each criterion. Having two people review the same sample now and then shows quickly whether the criteria are clear enough.

Regression testing when the prompt or model changes

A regression is something that used to work and now doesn’t. AI features regress more easily than most software, because a single change can affect every output at once. Rerun the full evaluation set whenever any of these change:

  • the prompt or system instructions, including small wording edits
  • the model, or the model version, including updates made by the provider
  • settings such as temperature or the maximum output length
  • the retrieval step, such as how documents are split, searched or ranked
  • the data the feature reads, such as a new price book or product catalogue
  • the code that validates or reshapes the output afterwards

Compare the new run with the last accepted run, case by case. A headline pass rate hides movement, because a version can fix six cases and break five and still look better on paper. Read every case that changed from pass to fail before you ship.

Where your provider allows it, call a specific model version instead of an alias that moves automatically. The model then changes only when you decide, and you can run the evaluation set first.

Monitoring and logging in production

An evaluation set tells you how the feature behaves on the cases you thought of. Production tells you about the rest. Log enough to reconstruct what happened on any single call:

  • the input and the output, or a reference to them where they hold personal data
  • the prompt version and the model version used
  • any documents or records retrieved and passed to the model
  • time taken, tokens used and cost
  • what the person did next, whether they accepted, edited, rejected or retried

The last item is the most valuable. A draft accepted unchanged is a strong signal of quality. A draft that is rewritten every time is a failure, even if nobody reports it. The screen design that makes this signal easy to collect is covered in designing AI features for when the answer is wrong.

Review a sample of real calls every week and turn the interesting failures into new evaluation cases. Set alerts for anything measurable, such as a rise in validation failures, slower responses or a higher cost per call, which we break down in what an AI feature costs per call.

Logs of AI calls often contain personal data, so treat them like any other personal data under UK GDPR. Keep only what you need, restrict who can read the logs, and delete them on a set schedule. The wider rules are in using AI with customer data under UK GDPR.

Adding AI to your product? We write the evaluation set and design the review screen alongside the prompt, before anything goes live. See how we build AI features.

Questions people ask

How many test cases does an AI feature need?

Start with around 50 real cases that cover both the common inputs and the awkward ones. Grow the set to a few hundred as the feature matures, adding every real failure you find. The spread of the cases matters more than the count.

Can we test an AI feature without real customer data?

You can start with realistic examples written by the team, but they tend to be cleaner than real inputs. Where you use real data, remove personal details the test doesn’t need, and check that your lawful basis and privacy notice cover the use.

What is an AI eval?

An eval is a repeatable test of an AI feature. It runs the feature against a fixed set of inputs and scores each output against agreed criteria, so the team can compare versions with numbers instead of impressions.

How often should we rerun the tests?

Rerun the full evaluation set before releasing any change to the prompt, model, settings, retrieval or output handling. Running it on a regular schedule as well helps catch changes on the provider’s side that nobody announced.

Written by Flavoureak

We build and run Estimark, CurioKids and CalcStack, and we build web and mobile software for other businesses. Start a project.