Operations
What is managed AI operations?
Every AI system picks up a second job on the day it goes live. Somebody has to keep it accurate, keep it safe and keep it inside its budget while the models underneath it are replaced, the data moves and the volumes grow. Managed AI operations is the name for that job. Here is what it covers, how often each part of it happens, what makes it cost what it costs, and the questions worth asking anyone who offers to do it for you.
- Author
- Luka Kokot, Co-founder and Chief Executive, Applicat AI
- Published
- Reading time
- 8 min read
Managed AI operations is the ongoing work of keeping an AI system accurate, safe and within budget after it goes live. It covers measuring the system's outputs against agreed cases, watching cost and latency, re-testing every change to the model, the prompts or the data before it reaches production, responding when something breaks, and reporting the results each month to the people who own the outcome. At Applicat AI it is the Run stage of the Applicat Method, delivered by the same engineers who built the system, and it is the part of an engagement that does not end.
The phrase names a job that exists whether or not anybody has been given it. A system that sorts claims or answers policy questions sits on top of models the AI labs replace every few months, data that changes weekly, and volumes that grow with the business. None of those changes announces itself. The upgrade is the risk sets out what moves underneath a live system and why that makes ownership unavoidable. This page is about the other half of it, which is what the owner actually does.
What the work consists of
Six things, and they are the same six whether the system reads documents, answers questions or takes actions on somebody's behalf.
- Scoring the system against your own cases. A set of real examples with agreed right answers, run on a schedule, so that accuracy is a number with a history behind it. That set is the evaluation suite, it belongs to you, and it grows every time the business meets an edge case the system handled badly.
- Watching what it costs and how long it takes. Cost per transaction and response time, tracked against the volumes the system was scoped for, with an alert when a request approaches a pricing threshold. A system can get more expensive while nothing about it changes, because the corpus it searches grew.
- Re-testing before a user meets the change. No new model, prompt, retrieval source or tool reaches production until the evaluation suite has run against it and the scores have been compared. These are the regression gates, and they separate a change you decided to make from one you find out about.
- Proving a new model on your work first. A better model arriving is a test run and a decision taken on the scores, then a staged rollout if the scores hold. Every model string in production is pinned, listed in a register with its published retirement date, and reviewed on a cycle, because the shortest of those dates is your real planning horizon.
- A way to stop it and put it right. A defined path to pause the system, roll back the actions that can be rolled back, tell the people affected and review afterwards, with named engineers on call and response times written into the contract.
- A written report every month. Accuracy, cost, volume, incidents and adoption, set against the number the system was built to move, sent to the person who owns that number. This is value reporting, and it is what makes next month's spend a decision somebody takes on evidence.
What turns that list into operations is the timetable. Each item is triggered by the calendar or by a change arriving, not by somebody noticing that an output looks wrong. Monitoring that begins when a user complains is a complaints process with a dashboard attached.
What drives the cost
Applicat AI gives indicative ranges in a first conversation and publishes no prices here, because the figure moves with four properties of the system itself. They are worth understanding before you ask anybody for a number, since they are also the levers.
- Volume, and what a wrong answer costs. How many decisions a month the system makes, and what happens when one of them is wrong. A system drafting internal summaries and a system releasing payments need different checkpoint design, different sampling and a different depth of reporting, and that gap accounts for most of the distance between a light engagement and a heavy one.
- How many cases are scored, and how often. A hundred cases re-run on every change is a different cost base from several thousand scored weekly with a person reviewing the disagreements. The right number is the smallest one that reliably catches the failures you care about.
- How often the ground moves. One model, one data source and a stable process needs less attention than three models with retrieval over a corpus that grows every week. Each model in production carries its own upgrade path and its own retirement date, and each of those is work somebody has to do.
- How much of the watching is automated. Scoring, alerting and the first draft of the monthly report can be built once and then run by the system itself. What stays manual is the judgment about what a drifting score means and what to do about it, which is why the fee tracks engineering attention more closely than it tracks transaction volume.
The commercial shape follows from that. Managed AI operations is a monthly arrangement with response times in the contract, reviewed at the end of each month against the report. Run carries the same kind of decision gate as the four stages before it in the Applicat Method: if the system has stopped being worth what it costs, you take it in-house with its documentation and its tests, and it carries on running.
Doing it in-house
An organisation with a platform team, an on-call rota and somebody willing to put their name against an accuracy number can do all of this itself, and some should. The requirements are the same either way: a named owner with the authority to pause the system, a test set drawn from your own cases that genuinely decides whether a change ships, a budget line that survives a busy quarter, and a report somebody senior actually reads. The failure case is almost never a team that tried this and found it hard. It is a system that went live, a project that closed, and nobody made responsible for the months afterwards.
What to ask for before you sign
- Ask to see the evaluation suite, and ask who writes the cases. If your own team supplies them and keeps them, the tests measure your work and they stay with you when the relationship ends.
- Ask what happens between the monthly reports. Daily watching and a monthly report are two separate commitments, and both should be named in writing.
- Ask what can reach production without passing the tests. The answer should be nothing.
- Ask how a model upgrade gets decided, and on whose evidence. A firm that cannot show you the comparison on your own cases is making that choice for you.
- Ask who is on call, by name, and what the contract says about response times and about pausing the system.
- Ask what happens the day you end it. The code, the data, the prompts and the tests should already be yours, and the system should keep running with nobody from the supplier left in it.
There is nothing exotic in any of this, which is part of why it gets skipped. The work has no launch date, no demonstration and nobody waiting on it, so it loses every argument about where the next engineer goes. It becomes visible in the quarter the numbers start sliding, and the useful question by then is how long the system has been getting it wrong, which is the one thing an unwatched system cannot tell you.