A few years ago, the most valuable person on many machine learning teams was the one who simply knew how to bring a model back to life. They knew which job to rerun first, which feature table tended to arrive late, which threshold to nudge when the numbers looked odd. Colleagues called them a wizard. Managers called them a risk. Today, the story across applied AI is quietly changing: the wizard is still respected, but the wizard’s knowledge is increasingly being turned into AI operations runbooks that anyone on the rota can follow.
This piece looks at where that shift stands, why it happened, and what the better teams are doing differently. It is a state-of-the-field note told through a composite story, not a survey with numbers, so treat it as an observation about habits rather than a measurement.
The morning the wizard was unreachable
Picture a small product team running a recommendation model for an online shop. The model is retrained every week, and one Tuesday the new version arrives with a strange symptom: it keeps suggesting the same few items to everyone. The engineer who built the retraining pipeline is on leave in the mountains. The rest of the team can read the code, but nobody has ever run the whole sequence by hand.
They lose most of a day. Somebody finds an old chat thread, somebody else guesses at the order of two steps, and a third person rolls back to the previous model, which turns out to be two versions old because a cleanup script had deleted the last good one. By evening the service is steady again, but everyone is rattled. Nobody wants to repeat that, and the team lead says what many leads have said in this situation: “We need to write this down.”
What changed in the way teams think about model upkeep
The early years of production AI were dominated by the building. Teams celebrated getting a model into service, and the care and feeding of it afterwards was treated as an afterthought, to be handled by whoever built it. As deployed models multiplied, that arrangement stopped scaling. A team with one model can hold its quirks in a single head. A team with a dozen cannot.
Three pressures pushed the field toward written procedures:
- Models are never finished. Data drifts, upstream tables change shape, and a model that was healthy in spring can be quietly wrong by autumn. Upkeep is a permanent duty, not a project.
- Rotas replaced heroes. Many organisations now share on-call duty across engineers who did not build every system. They need instructions, not tribal memory.
- People move. Specialists change companies and teams. When they leave, the unwritten parts of the job leave with them.
The anatomy of a good retraining runbook
The runbooks that hold up in practice are surprisingly plain. They do not explain the mathematics. They describe a sequence a tired person can follow at an inconvenient hour. A typical retraining runbook contains:
- The trigger. Is this a scheduled retrain, or a response to a symptom such as falling accuracy on recent examples?
- The preconditions. Which input tables must be fresh, which credentials are needed, and where the last known-good model is stored.
- The steps. Start the data preparation job, confirm it produced the expected number of partitions, launch training, wait for the evaluation report, compare it with the current model on a fixed holdout set.
- The promotion rule. A written, boring criterion for when the new model replaces the old one, and who signs off.
- The way back. Exactly how to restore the previous version, including a reminder never to delete it until the next one has proven itself.
- The hand-off note. Where to record what was done, so the next person starts from facts.
Notice what is missing: cleverness. A runbook is a deliberately unglamorous document, and its value is that the step after “launch training” is never a matter of opinion.
Where the practice is strongest, and where it lags
In my reading of how teams talk about this, the habit is strongest around anything that has bitten people before. Rollbacks are documented more often than retraining. Data refresh checks are documented more often than evaluation. Access requests, the dull business of getting a new teammate permission to read a feature store, are documented least of all and cause more hidden delay than anyone admits.
The weakest area is judgement-heavy work, such as deciding whether a drop in a metric is a real failure or ordinary noise. Teams often assume this cannot be written down. The more thoughtful ones disagree: they cannot capture the judgement itself, but they can capture the questions an experienced person asks first. Did the input volume change? Did a source system release something this week? Is the drop confined to one segment? A short list of opening questions turns a vague worry into a structured investigation.
The same applies to business use of models. Articles such as this look at how AI supports better business decisions describe the upside, but the upside depends on someone keeping the model dependable week after week, and that is exactly what runbooks are for.
A short story about handing over a model
Consider the same shop team six months later. They have written four runbooks: weekly retraining, emergency rollback, feature-table delay and new-joiner access. The engineer from the mountains is back, and then, three months on, takes a role at another company.
The hand-over takes an afternoon instead of a month. A junior colleague follows the retraining page with the leaver sitting beside them, taking notes where the page falls short. By the end the page has two new warnings and a corrected command. The leaver says it was strange to see their own habits read back to them as numbered steps, and a little humbling to realise how much they had been carrying around unrecorded.
That experience mirrors what many teams report: writing the runbook is a gift to the future, but it is also a mirror. It exposes the places where the system depends on luck.
Speed matters too: runbooks for performance-sensitive systems
Not every AI system is a weekly batch job. Systems that handle images or video in real time have their own recurring chores, from warming caches to checking that processing queues are draining. A piece on scaling fast visual pipelines without choke points shows how much operational detail sits behind a smooth demo. Each of those details is a candidate for a short written routine, because performance problems tend to strike at awkward hours and reward calm, methodical responders.
What the better teams are doing now
Across the teams that seem to be doing this well, a few habits recur:
- They write runbooks after real events. Every incident or near miss ends with a small edit to the relevant page.
- They test with newcomers. A runbook that only its author can follow is a note to self, not a runbook.
- They keep pages short. One task, one page, one owner.
- They date everything. Readers need to know when a page was last proven to work.
- They rehearse. Occasionally, a planned drill has someone other than the author run a rollback in a safe environment.
- They keep a reading habit. Engineers look outside their own shop for ideas on how to run data and model workflows more steadily.
On that last point, practitioners often browse blogs such as TheDataOps.org, which follows the path from DevOps practice towards dependable data operations. Reading how neighbouring disciplines tame their own recurring chores tends to spark fresh ideas for model upkeep.
Where this goes next
It is reasonable to expect that more of the routine will be automated, and that is a good thing. But automation does not remove the need for runbooks; it changes what they say. A script that restarts a pipeline still needs a page describing when to run it, what to check afterwards and when to stop and ask for help. Teams that skip that page tend to discover the gap during their first serious failure.
The deeper change is cultural. The field is slowly moving from admiring the person who can fix anything to admiring the team whose work does not depend on any one person. That is a healthier definition of expertise, and it makes AI systems easier to trust, easier to hand over and kinder to the people who look after them.
If your team has only ever kept this knowledge in one head, start small. Choose the one model task that would cause the most trouble if its keeper vanished, write it as numbered steps, and have someone new run it. That single page is the beginning of an operation that no longer needs a wizard.