Backend Foundations for AI Products: Data, Queues and Control

In the AI products we have built and the ones we have been called in to rescue, the model is rarely the thing that broke. The failures are ordinary backend failures, made sharper by three properties that model calls have and typical backend operations do not: they are slow, they are expensive, and they sometimes fail in ways that look like success.

Here is what that changes about the system around the model.

Long operations do not belong in the request cycle

A model call can take seconds. Sometimes tens of seconds. A chain of them can run for minutes. Handling that inside an HTTP request produces a familiar set of problems: gateway timeouts at thirty or sixty seconds, retries that silently duplicate expensive work, a mobile client on a train that loses the connection and loses the result with it.

The pattern that works is the one long-running systems have always used. Accept the request, persist a job record, return an identifier immediately. Process in a background worker. Let the client poll or subscribe for the result. Stream partial output where it improves the experience.

This is not exotic infrastructure, but it is a decision that has to be made early, because retrofitting it means changing every client that assumed a synchronous response. The API contract, the mobile app, the error handling, all of it.

Idempotency is not optional when calls cost money

Retries are a fact of distributed systems. Networks drop, workers restart, users double-tap. In conventional software a duplicated request writes the same row twice and you clean it up. When the operation is a model call, every duplicate is money, and a job queue with an at-least-once delivery guarantee will deliver duplicates.

So every expensive operation needs an idempotency key, derived from the input, or supplied by the client — checked before the work is done and recorded after. Take the same request twice, do the work once, return the same answer.

The same mechanism doubles as a cache. If two users ask an identical question, or one user asks twice, you already have the answer stored. That is the cheapest cost saving available and it falls out of doing idempotency properly.

Every model call needs a budget and a ceiling

A conventional backend degrades under load: things get slow, queues back up, someone gets paged. An AI backend under the same load keeps working and bills you for it. There is no natural back-pressure, because the provider is happy to serve every request.

So the limits have to be explicit, and they have to exist in the code rather than in someone's intention:

• Per-user and per-account rate limits, applied before the expensive call, not after.
• A hard cap on chain length. Any loop where the model decides whether to continue needs a maximum iteration count. Without it, one malformed input can spin until someone notices.
• Timeouts on every call, with a defined behaviour when they fire.
• A circuit breaker for provider outages, so a degraded provider does not turn into a queue of retries that all fail expensively.

We go deeper on the economics of this in shipping AI features without wrecking your unit economics; this is the enforcement layer for it.

Model outputs are untrusted input

This is the mental shift most teams make too late. A model's response is not a return value from your own code: it is closer to a form submission from a stranger. It usually has the right shape. Occasionally it does not.

Which means the boundary needs the same treatment any untrusted input gets. Validate the structure against a schema before anything downstream touches it. Have a defined path for when validation fails: retry once, fall back, surface an error, rather than letting a malformed response propagate. Never interpolate model output into a query, a shell command, or a template that executes.

That last one deserves emphasis when the model is summarising documents that users upload. Content written by a third party, passed through a model, and acted on by your system is a genuine injection path, and it is tested for far less often than it should be.

Store the whole interaction, not just the answer

The instinct is to store the output and discard the rest. Six weeks later, when a customer reports a bad result, you have the output and no way to explain it.

Persist enough to reconstruct any request: the input, the prompt version, the model and parameters, what was retrieved if anything, the raw response, token counts, latency, and cost. This is one table, and it is the single highest-use thing you can build early.

It is what turns "the AI is being weird" into a specific, reproducible case. It is where the evaluation set comes from, real failures beat invented test cases. It is how you answer whether last week's change helped. And it is how you find out that quality dropped because the provider updated the model behind a stable name, without a line of your code changing.

If you cannot reconstruct exactly what produced a bad answer, you cannot fix it, you can only change something and hope.

Do not marry a provider you cannot leave

Provider APIs look similar enough that teams call them directly from feature code, and then discover the cost of that when they want to move — to a cheaper model for simple tasks, to a second provider for redundancy, or away from one whose terms changed.

A thin internal interface is enough. Your application asks for a completion with a task name and inputs; behind that, one place decides which provider and model serves it, applies the timeout and retry policy, and does the logging. That indirection costs an afternoon and buys three things worth much more than it: routing easy tasks to cheaper models without touching feature code, failing over during an outage, and A/B testing a model change on a fraction of traffic.

The failure mode to avoid is the opposite extreme: a heavyweight abstraction attempting to normalise every provider's capabilities. That ages badly, because the interesting features are exactly the ones that differ. Keep it thin, and let it leak deliberately where you need something specific.

Streaming changes the shape of the endpoint

If your interface streams output as it is generated, the backend contract changes more than it first appears, and a few things routinely get missed.

You still need the complete response persisted server-side once generation finishes: for logging, cost accounting and the eval set. A client that assembled the text does not help you when it disconnects halfway.

You need a defined behaviour for a stream that fails mid-flight. The user has already read three paragraphs; the fourth never arrives. Silently stopping looks like a finished answer that trails off, which is worse than an error.

And validation gets harder, because you cannot schema-check output you have not finished receiving. Where output must satisfy a contract, either stream a non-structured part and validate the structured part separately, or accept that validation happens at the end and design the interface for a late failure.

None of this is a reason to avoid streaming, it is a better experience for long generations. It is a reason to treat it as an architectural decision rather than a display option.

Prompts are code, so version them

A prompt sitting in a string literal, edited in production to fix an issue, is an undocumented change to system behaviour with no review, no history, and no way to roll back.

Prompts belong in version control, changed through the same review process as any other code, with a version identifier recorded against every request that used them. Then a quality regression is traceable to a specific change on a specific date, which is the difference between a ten-minute fix and a week of archaeology.

The data model still decides everything

The oldest lesson in backend engineering survives contact with AI entirely intact: a generative layer over a confused schema produces confident nonsense.

If your customer records are duplicated across three tables with no clear owner, retrieval will surface all three and the model will blend them into an answer that is fluent and wrong. If permissions are enforced in the UI rather than the data layer, retrieval will happily fetch a document the user should never see and quote it back to them. That second one is not a quality bug, it is a data breach, and it is the failure mode we see most often in retrieval systems built quickly.

Getting the schema and the permission model right is unglamorous work that no demo shows off. It is also the work that determines whether the impressive demo becomes a product.

The short list

Before the model call is interesting, these should be true:

• Long operations run in the background, with a job record the client can poll.
• Expensive operations are idempotent, and the idempotency store doubles as a cache.
• Rate limits, chain-length caps, timeouts and a circuit breaker exist in code.
• Model output is schema-validated at the boundary, with a defined failure path.
• Every interaction is logged in full, inputs, versions, cost, latency.
• Prompts are versioned in the repository and stamped on every request.
• Permissions are enforced in the data layer, below anything retrieval can reach.

None of it is novel. All of it is what separates an AI demo from an AI product, and it is the substance of what our backend engineering work consists of.