The symptom looks harmless at first. An anomaly detector that used to flag real incidents begins to flag a few too many ordinary evenings. Nobody files a ticket, because nothing is on fire. Three months later the on-call team has quietly learned to ignore the model, and the one genuine signal in a month is buried. This is the usual shape of AIOps trouble: slow drift, not sudden failure. The remedy that tends to work is almost embarrassingly modest, which is ten minutes of focused review every day.
Diagnosis: why drift goes unnoticed
Three conditions let drift hide. First, the model is judged by an average that moves slowly, so no single day looks wrong. Second, the people who could notice are busy with the incidents the model did catch. Third, review is treated as a quarterly project, which means the inputs have changed several times before anyone looks.
Evidence from ordinary operations supports this pattern. Traffic mixes shift when a product launches. A service is renamed and its old metric series goes silent. A deployment tool starts attaching new labels. Each change is tiny, yet a model trained on last season’s shapes treats the new shapes as strange. The problem is therefore less about the algorithm and more about the gap between how often the world changes and how often a human glances at the output.
The fix: small daily reps, not big audits
Instead of a heavy review, schedule a ten-minute sitting each working day with one narrow question. The narrowness matters. A broad question (“is the model healthy?”) invites a shrug. A narrow one (“what did it flag yesterday that no human acted on?”) has a definite answer you can write down.
Rotate through a short list of narrow questions across the week:
- Monday: pull yesterday’s alerts that nobody acknowledged and read three of them.
- Tuesday: check whether any input series has gone flat or vanished.
- Wednesday: compare one alert with the raw metric behind it and ask whether a person would have called it unusual.
- Thursday: look at one recently changed service and see whether its alerts shifted.
- Friday: write one sentence on what you learned and one thing to try next week.
Ten minutes is not enough to retrain anything, and that is intended. The sitting only produces observations. Observations accumulate in a running note, and the note tells you when a heavier intervention is justified.
Reading the evidence without fooling yourself
An analytical habit needs a guard against wishful reading. When you inspect an alert, write your verdict before you look at what actually happened afterward. “Looks real” or “looks routine” takes seconds, and over a few weeks the ratio of correct calls becomes a crude but honest measure of the model’s usefulness. If you change your verdict after seeing the outcome, you have only learned what hindsight feels like.
Keep each day’s record tiny: date, the thing you checked, your verdict, a short reason. A spreadsheet with four columns is plenty. The temptation is to build a dashboard, but a plain table you actually update beats an elegant one nobody opens. When the note shows the same cause three times, such as an unlabeled new service, that is your cue to act on the cause itself.
Where decision-making articles help
It also helps to remember why anyone trusts a model in the first place. Writing on the role AI plays in sharpening commercial decisions frames models as inputs to human judgement, not replacements for it. That view fits the daily review: the ten minutes are the moment where human judgement rechecks the model’s input to the team.
For readers who are newer to operations work and wonder what to study alongside this habit, a general overview of the skills involved in becoming a DevOps engineer gives context on the tooling that produces the metrics your detectors consume.
A concrete walk-through
Imagine a team that runs a detector on request latency. In the first week of the habit, the Monday review shows that most unacknowledged alerts cluster around the hour when a nightly batch job starts. The Tuesday check finds nothing missing. By week three the Friday notes show the same hour appearing again and again.
The diagnosis is now evidence-based: the model has never been told that the batch window is expected. The fix is correspondingly small. The team adds the batch window as a known pattern, watches the same Monday list for a week, and sees the unacknowledged alerts fall. No grand project was needed, and the cost was roughly an hour of cumulative review spread over three weeks.
Where a self-assessment can fit in
Teams sometimes ask how to decide which area to review first. One option is to browse the questionnaire and library on OpsRoadmaps, an engineering-operations site that offers a self-assessment and a collection of roadmaps, checklists and templates, then borrow a checklist as the seed for the weekly list of narrow questions. Treat anything you find there as a prompt for your own judgement, not a verdict about your team.
Failure modes of the habit itself
Even a ten-minute routine can decay, and it is better to name the common failure modes in advance:
- The sitting grows to an hour. Cut it back. A habit that costs an hour will be skipped by Thursday.
- Nobody reads the notes. Put the Friday sentence somewhere the team already looks, such as the channel where incidents are discussed.
- The same person always does it. Share the rota, so a single absence does not stop the practice.
- Findings are never acted on. Reserve one small slot per fortnight for fixing the top cause from the notes.
A note on ownership
Give the daily sitting a named owner for each week, even if the role rotates. Without an owner, a shared habit turns into everyone’s good intention and nobody’s calendar entry. A rotating owner also spreads knowledge: after a quarter, most of the team has read raw alerts and formed their own opinion of what the detector gets right, which is worth more than any single report.
What to measure after a month
Resist elaborate metrics. After four weeks, ask just three things. How many of the alerts you reviewed would a person have called real? Did the count of ignored alerts fall? And did anyone on the team learn something about the system that they did not know before? Should that third answer be a firm yes, the routine has repaid its cost already, whatever the percentages say.
Slow drift is a patience problem. The model does not fail on a Tuesday; it wanders away from reality by degrees. Ten minutes of attention a day is a small enough step to take on the busiest week of the quarter, and repeated often enough it keeps the model and the people who rely on it pointed at the same world.