Teams running machine learning in production return again and again to a short set of questions about how to react when something breaks. The answers tend to share a shape: decide the response before the incident, write it down, and keep it small. This article works through those questions one at a time, treating if-then plans for AI systems as a practical discipline rather than a slogan.
What is an if-then plan in an AI operations setting?
It is a pairing of a specific, observable trigger with a single pre-agreed response. “If the input data for the scoring model arrives with a missing column, then the pipeline stops and the owner is paged.” The wording is deliberately dull. A dull rule is a rule that people follow at three in the morning.
The analytical point is that most operational delay is not caused by difficulty. It is caused by deliberation. When something breaks, a group of capable people spend ten minutes discussing which of five reasonable options to try first. A written plan converts that discussion into an action that was already agreed on a calm day.
Why do models need triggers at all if monitoring already exists?
Monitoring reports that a number moved. Nothing in it prescribes the next step. Many teams have dashboards full of lines and no shared answer to “so what happens now?”. The plan supplies the missing second half of the sentence.
Evidence supports keeping the pairing narrow. A trigger such as “accuracy looks worse” is too loose to test. A trigger such as “the weekly evaluation on the held-out sample falls below the floor we agreed” can be checked by anyone and argued about by no one. The more measurable the trigger, the less room there is for interpretation under pressure.
Which triggers are worth writing first?
Start with the failures that have already happened to you. Review your last few incidents and ask which of them would have been shorter if the response had been decided earlier. Typical candidates include:
- If the model serves a request that exceeds the agreed latency limit repeatedly, then traffic shifts to the previous stable version.
- If the feature pipeline delivers stale data, then the model falls back to a cached prediction and flags it as stale.
- If the evaluation set shows a clear drop after a retrain, then the release is blocked and the previous artifact stays live.
- If a prompt or output filter triggers on the same pattern twice in a row, then the owner reviews the sample before the next release.
Four plans are plenty to begin with. Teams that start with fifteen rarely maintain any of them.
How specific should the response be?
Specific enough that two different engineers would do the same thing. “Investigate” is not a response; it is a postponement. “Switch the alias to the prior version and open a ticket” is a response. A reasonable test is to hand the plan to a new colleague and see whether they can act without asking a question.
It also helps to name the decision owner in the plan. The if-then statement says what happens; a short line beside it says who authorises exceptions. Without that, a rule can stall the moment someone wonders whether they are allowed to follow it.
Does this reduce the need for judgement?
No. It relocates judgement to a better time. The thinking happens when you write the plan, with data in front of you and no alert sounding. In the moment, people execute, and afterwards they judge whether the plan was good. That sequence is more rigorous, because it creates a record you can compare against outcomes.
A useful habit is to log each time a plan fires, and whether it helped. After a few weeks you can see which plans never trigger, which trigger constantly, and which triggered and made no difference. Those three groups tell you what to prune, what to fix at source, and what to rewrite.
How do we avoid plans that nobody reads?
Place them where the work happens. A plan buried in a wiki page is a plan that will be rediscovered during the next outage. Attach the written response to the alert itself, so whoever is paged sees the trigger and the action in one view. Keep each plan to a few lines. If you find yourself writing paragraphs, you are documenting a procedure, which is a different job.
Review the list on a schedule. Plans decay as systems change. A response that referred to a retired service is worse than no response, because it lends false confidence.
What counts as good evidence that the approach is working?
Look at the time between detection and the first corrective action, not at the number of plans. If that gap shrinks and incidents stop reopening, the plans are earning their place. Resist the temptation to measure success by how many rules exist, since that rewards bureaucracy.
Qualitative signals count too. Do on-call engineers report fewer arguments in the incident channel? Do new joiners feel confident acting during their first rotation? Those observations are anecdotal, but they point toward the same outcome the numbers do. For a broader view of how analytical reasoning feeds operational decisions, an overview of how AI supports business decision making covers the reasoning side, while a plain explanation of image classification shows how model behaviour can be reasoned about before it ever reaches production.
What about the people who travel for work?
Engineers who support global systems sometimes find themselves on call from airports and hotels. The same logic applies to staying reachable: settle in advance how you will respond when the primary connection drops. One option some travellers consult is a website that compares SIM and eSIM options for travelers, which organises country, provider, plan and guide pages so the choice can be made before departure rather than at the arrivals gate.
Where should a team begin this week?
Pick the single incident that annoyed you most last month. Write one trigger and one response for it, attach the owner, and link it from the alert. Run it for a fortnight. If it fires, review it; if it never fires, check that the trigger can actually be observed. Then add a second plan.
Nothing about this is elaborate, which is exactly why it holds up. A short list of specific, written, regularly reviewed if-then plans gives an AI operations team something that dashboards alone cannot: a response that is already decided when the pressure arrives. Keep each rule testable, keep each owner named, and let the logs, not opinions, tell you what to change next.