# Testing AI Features: Evals, Regressions and What QA Becomes

> Traditional tests assert exact outputs, which non-deterministic features cannot provide. What replaces them, and why the alternative is not "test it manually".

**Published:** 2026-08-06

**Tags:** Testing, AI Engineering, Quality

---

Ask a team how they test their AI feature and the honest answer is usually some version of "we try it and it seems good." Not because the engineers are careless — because the tools they know do not apply. `assertEquals` needs a right answer, and a summary has no single right answer.

What replaces exact-match testing is not manual checking. It is a different kind of automated test, and it is buildable in a week.

## Why the usual tools stop working

Three properties break conventional testing at once.

**The output is not deterministic.** Same input, different wording. An exact-match assertion fails on a perfectly good answer, so it gets deleted, and coverage quietly disappears.

**Quality is a matter of degree.** A summary can be accurate but too long, fluent but missing the key point, correct but in the wrong tone. There is no boolean.

**Failures are plausible.** This is the dangerous one. Conventional software fails loudly: it throws, returns a 500, renders nothing, and monitoring pages someone. A model-backed feature fails by producing something well-formed, confident and wrong. Error rates stay flat. Nothing alerts. The only signal is a user who noticed, and most users do not report it; they just trust the feature a little less, then stop using it.

## What an evaluation set is

An eval set is the AI equivalent of a test suite: stored inputs, a definition of what good output looks like for each, a way to score an actual output against that, and a command that runs the lot and prints a number.

It is deliberately mundane, and its value is almost entirely in existing *before* you start changing things. With one, a prompt change or a model upgrade is a measured decision. Without one, every change is a coin flip defended in a meeting by whoever tried the most examples by hand.

**Where the cases come from matters more than how many there are.** Fifty real inputs beat five hundred invented ones. Pull them from what users submitted, which is why the full request logging described in [backend foundations for AI products](/blog/backend-for-ai-products) is a prerequisite rather than a nicety. Every reported bad output becomes a permanent case, exactly like a regression test.

## Four ways to score, cheapest first

**Assertions on structure.** Not everything is fuzzy. Did it return valid JSON? Are the required fields present? Is the label one of the four allowed values? Is it under the length limit? These are ordinary deterministic tests and they catch a great deal: run them first because they are free.

**Required and forbidden content.** Does the answer contain the figure it was supposed to extract? Does it avoid naming a competitor, or promising a refund? Keyword and pattern checks are crude, fast, and catch the failures that matter commercially.

**Model-graded scoring.** Use a capable model as a judge: given the input, the expected qualities and the actual output, score it. This works, with two caveats worth stating plainly. Judges are biased toward longer, more confident answers, so calibrate against human judgement on a sample before trusting the number. And a judge is itself a model call, so a large eval set has a real running cost.

**Human review, sampled.** The ground truth, and too slow to run on every change. Use it periodically on a sample, and use it to check that your automated scores still agree with what a person thinks. When they diverge, the scoring method is wrong, not the people.

> The goal is not a perfect score. It is a number that moves in the same direction as quality, so you can tell whether a change helped.

## Run it as a regression suite

The eval set earns its cost when it runs automatically, on every prompt change, every model or parameter change, every retrieval change, and on a schedule.

That last one is not padding. Providers update models behind stable API names. Your output can change materially without a single line of your code changing, and a weekly scheduled run is how you find out in days rather than discovering it through a support ticket. Treat a score drop like a failing build.

## The test class people skip

Adversarial inputs deserve a standing place in the suite, not an occasional security review.

If your product summarises documents that users upload, then text written by a stranger reaches your model. Instructions embedded in that text, telling the model to ignore its instructions, reveal its prompt, or produce something harmful — are a live injection path, and the equivalent of SQL injection for this generation of software.

Keep a permanent set of these cases: instruction overrides, attempts to extract the system prompt, inputs designed to make the model exceed its remit, and the boring ones too, empty input, enormous input, the wrong language, malformed encoding. Then assert on behaviour: the output contract still holds, the system prompt is not disclosed, the model declines what it should decline.

## Test the retrieval separately from the answer

If your feature retrieves before it generates, a bad answer has two possible causes, and conflating them wastes a great deal of time. Either the right material was never fetched, or it was fetched and the model used it badly. These need different fixes and different tests.

Score retrieval on its own terms, before the model is involved. For a set of real questions, record which documents *should* be found, then measure two things: whether the right material appears at all, and how much irrelevant material comes with it. The second matters more than teams expect, precision is usually the binding constraint, because burying the right passage among twenty wrong ones degrades the answer and costs tokens on every request.

Then score the generation step *given correct context*: hand the model the right material deliberately and check what it does with it. If retrieval scores well and the answers are still poor, the problem is the prompt. If retrieval scores badly, no prompt work will help.

One case deserves its own test because it is where retrieval systems embarrass themselves: **the question with no answer in the corpus.** The desired behaviour is to say so. The default behaviour is to synthesise something plausible from whatever was nearest. Keep a permanent set of unanswerable questions and assert that the system declines.

## How many cases, and what to do when scores disagree

Two questions that come up immediately.

**Size.** Start at twenty and grow. Twenty real cases with carefully considered expected outputs beat two hundred casual ones, because the value is concentrated in the thinking that produces the expectation. Grow through failures rather than through generation: every genuine bad output that reaches a user becomes a permanent case, so the set drifts toward the things that go wrong with your product.

Keep it representative rather than balanced. If eighty percent of real traffic is one straightforward request type, an eval set of exclusively hard edge cases will report a low score that has nothing to do with what your users experience.

**Disagreement.** Sooner or later the automated score says a change improved things and the team says it made them worse. When that happens, the score is wrong. Not the people. A judge scoring on the qualities you defined has been given the wrong qualities, or is rewarding length and confidence over usefulness, which is its most common bias.

The fix is to treat calibration as ongoing work: periodically have a person score a sample blind and compare against the automated score. Where they diverge, the definition of good needs rewriting. An eval score nobody trusts is worse than no score, because it gets cited in decisions and then quietly ignored.

## What conventional QA still owns

None of this replaces normal testing. In every AI product we have built, most of the engineering is ordinary software, and it needs the testing it always needed, unit tests, integration tests, and end-to-end coverage of the flows around the AI feature.

Two of those flows are specific to this kind of product and get missed: what the interface does while a slow generation is running, and what it does when the model call fails or returns something invalid. Both are common in production and both are frequently untested, because they are hard to trigger by hand. They are not hard to trigger in a test.

## Starting from nothing this week

- **Log every interaction in full.** Without this there is no source of real cases.

- **Collect twenty real inputs and write what a good output looks like for each.** Do this with whoever owns the feature commercially, not only with engineers — the disagreements that surface here are the most valuable output of the whole exercise.

- **Add the structural assertions.** Free, fast, and they will already catch something.

- **Add model-graded scoring** for the qualities structure cannot check.

- **Wire it into CI and a weekly schedule.**

- **Feed every reported failure back in**, permanently.

A week of work, and it converts your AI feature from something you hope is working into something you can make claims about. That is the difference our [software testing and QA](/services/software-testing) practice is built around.

_Read online: https://mobizio.io/blog/testing-ai-features-evals_