# Prototype to Production: The AI Readiness Checklist

> The demo works. That is roughly a third of the way. The concrete list of what stands between an impressive prototype and something you can charge for.

**Published:** 2026-08-04

**Tags:** AI Engineering, Product Strategy, Consulting

---

The demo works. Someone showed it to the team, the room was impressed, and there is now a date in a plan.

This is the most consistently underestimated moment in AI product work. A prototype that produces a good answer for a prepared input has cleared roughly a third of the distance. What remains is not polish: it is the difference between something that works when you drive it and something that works when strangers do.

Here is that remaining work, as a checklist rather than a warning.

## Quality: can you prove it, or only demonstrate it?

- **An evaluation set built from real inputs**, not invented ones, with a defined notion of good output for each case.

- **A score you trust**, checked against human judgement on a sample.

- **The eval running in CI** on every prompt, model or retrieval change, and on a schedule to catch provider-side model updates.

- **A known failure rate.** Not "it works well": a number, and a shared view on whether that number is acceptable for this feature.

- **Failures fed back in permanently**, so the same regression cannot ship twice.

If the prototype has none of these, this is the first block of work, not the last. [Testing AI features](/blog/testing-ai-features-evals) covers how to build it from nothing in about a week.

## Cost: does the feature survive its own success?

- **Cost per invocation measured** at the model and context size you will ship.

- **Expected invocations per user per month**, including the heaviest one percent.

- **That multiplied out and compared against revenue per user.** If it loses money, something changes before launch: pricing, limits, or the design.

- **Per-request cost logging** by feature and account, because a monthly provider total is the wrong granularity to act on.

- **Rate limits enforced in code**, especially on any free tier.

- **A ceiling on chain length** anywhere the model decides whether to continue.

## Reliability: what happens when things go wrong?

- **Long operations moved out of the request cycle**, with a job record the client can poll.

- **Idempotency on every expensive operation**, so a retry does not double the bill.

- **Timeouts on every model call**, with defined behaviour when they fire.

- **A defined path for invalid output**, retry, fall back, or surface an error. Not "assume the schema held".

- **A provider outage plan.** A second provider, a degraded mode, or an honest error message. Deciding this during an outage is the expensive way.

- **Monitoring on quality, not just uptime.** A model-backed feature fails while returning HTTP 200, so uptime dashboards show green through the entire incident.

## Safety: what can a stranger make it do?

- **Prompt-injection cases in the test suite**, if any user-supplied or third-party content reaches the model.

- **Permissions enforced in the data layer**, below anything retrieval can reach. Retrieval that fetches a document the user should not see and quotes it back is a data breach, not a quality bug.

- **Model output treated as untrusted input**, never interpolated into a query, command, or executing template.

- **A clear position on what goes to the provider**, and whether your terms and privacy policy say so. This one is not optional and it is not only an engineering question.

- **A refusal path** for requests the feature should decline, tested.

- **Rate limits that also bound abuse**, not just cost. An endpoint that runs an expensive model call on unauthenticated input is a denial-of-wallet attack waiting to be found.

- **A decision on training data.** Whether your customers' content may be used to improve anyone's model, stated somewhere a customer can read it. Enterprise buyers ask this early and a vague answer loses deals.

## The interface: does it behave while it thinks?

- **A designed waiting state.** An eight-second generation is a different feature from an instant one, and a spinner is not a design.

- **Streaming where it helps**, so the user sees progress rather than a gap.

- **An error state that tells the user what to do next**, not a toast saying something went wrong.

- **A way to report a bad output**, wired to somewhere a human looks — this is also your best source of eval cases.

- **Honesty about what it is.** Users are markedly more forgiving of a system that is upfront about being generated and occasionally wrong than one that presents its output as authoritative.

## Operations: can someone other than the author run it?

- **Prompts versioned in the repository**, changed through review, and stamped on every request.

- **Full interaction logging**, input, prompt version, model, parameters, retrieved context, raw output, tokens, latency, cost.

- **A runbook for the three failures you will get**: provider outage, quality regression, cost spike.

- **Someone who owns the quality number** after launch. Quality drifts, and drift with no owner is how a good feature quietly degrades.

- **A model-upgrade procedure**, because the model you launch on will not be the one you run in a year.

## Launch in stages, not at once

Every item above is easier to satisfy if the launch is graduated rather than a switch. Three mechanisms do most of the work, and all three are cheaper to build before launch than during an incident.

**A kill switch.** A configuration flag that disables the AI feature and falls back to whatever the product did before it, changeable without a deploy. Ten minutes of work. It converts "we have a serious quality problem" from an emergency release into a decision someone can make calmly.

**Percentage rollout.** Enable for five percent of accounts, then twenty-five, then everyone. Cost and quality problems announce themselves at five percent, where they are affordable and the affected group is small enough to contact individually.

**A named early group.** Twenty accounts who know they are early, have a direct line to you, and expect roughness. They report the failures that never reach your logs as complaints: the quiet disappointments that otherwise show up months later as churn.

The reason to build these into the plan rather than hold them in reserve is that the failures this list guards against are usually gradual. Cost creeps. Quality drifts. A staged rollout is what turns a slow problem into an early signal.

## What to do with the reds

Walking the list produces a set of unfinished items and a natural instinct to fix all of them before shipping. That is usually the wrong call, and it is worth having the sorting conversation explicitly.

**Genuinely blocking**, anything under Safety, plus a known failure rate and a hard cost ceiling. These are the items where the consequence of being wrong is a breach, a bill you cannot pay, or a claim you cannot support. They do not trade against a date.

**Manageable behind a limit**, much of Cost and Reliability. A feature with unproven economics can ship to a small paying cohort with a low rate limit. A missing provider-failover plan is survivable at low volume with an honest error message. These are reds you can hold while the exposure stays small.

**Genuinely deferrable** — parts of Operations and the interface. A runbook can be written in the first month. Streaming can wait. A model-upgrade procedure is not needed on day one.

What separates the first group from the third is not importance, it is whether being wrong is recoverable. That is the question to run each red item through, out loud, with the people who own the launch date in the room.

## What this is really telling you

Read the list in one pass and the pattern is hard to miss: almost none of it is about the model. It is logging, limits, permissions, error paths, ownership, ordinary engineering discipline applied to a component that happens to be probabilistic.

That is why the prototype-to-production gap surprises people. The hard part looked finished, so the remaining work looks like formality. It is not formality. It is most of the build, and skipping it is the most reliable way to launch something that erodes trust faster than it earns it.

> The prototype proves the idea can work. Everything on this list is what makes it work for people who are not you.

## How to use it

Do not treat this as a gate to pass in order. Walk it once with the team and mark each line green, amber or red. The reds are your remaining scope, and they are usually clustered: a team with no logging typically also has no eval set and no cost visibility, because they all come from the same missing foundation.

Then make the honest call about the launch date, with the list in front of everyone rather than in one engineer's head. That conversation is most of the value of writing it down, and it is where our [software consulting](/services/software-consulting) engagements usually begin. If you want the underlying reasoning rather than the checklist, [what AI-native product engineering changes](/blog/ai-native-product-engineering) sets out the principles these items come from.

_Read online: https://mobizio.io/blog/ai-production-readiness-checklist_