Teams that run machine-learning systems in production tend to remember the outages and forget the scares. A model that drifted for a week before someone noticed, a retraining job that quietly used stale data, a threshold that almost went live with the wrong value: these close calls rarely make it into a postmortem because nothing broke. Yet they carry the same information as a real failure, at no cost to customers. Five persistent myths keep teams from using that free information in AI operations, and each deserves a precise correction.
Myth 1: “If nothing broke, there is nothing to learn”
The absence of an outage tells you about the outcome, not about the system. A prediction service that served a degraded model for ninety minutes and recovered on its own has exposed a missing alarm, whether or not any user complained. The same applies to a feature pipeline that produced nulls overnight and was patched at breakfast.
The correction is to separate two questions. Did harm occur? And what was the distance between the event and harm? In many AI operations incidents the distance was a single lucky observation: an engineer happened to open a dashboard, a downstream consumer happened to raise a query. Luck of that kind is a finding. Record it with the same seriousness you would give an incident, and ask which control should have caught it without any luck at all.
A useful habit is to attach a rough label to each scare: caught by design, caught by habit, or caught by chance. Only the first label is comforting. If most of your entries land under the third, your real protection is the schedule of whoever happens to be awake, which is not something you can put in a runbook or promise to a stakeholder.
Myth 2: “Near misses are just small incidents, so our incident process covers them”
Incident processes are built around impact: severity levels, customer communication, on-call escalation. A near miss has no impact, so it usually fails the entry criteria and never gets a ticket. Teams then conclude, wrongly, that the process is comprehensive.
A lightweight parallel channel works better. Keep a short form with four fields: what nearly happened, what stopped it, how far it was from harm, and what would have made the catch automatic. Review the entries in a fortnightly session that is explicitly not an incident review. Nobody is on trial and no severity is assigned. The point is to harvest patterns, such as three separate scares involving the same data source, that no single ticket would reveal.
Keep the tone of the form matter-of-fact, too. Phrases such as “almost caused” and “was a few minutes from” give later readers a clear sense of scale without drama, and they make it easier to compare entries written by different people on different days.
Myth 3: “Models fail in exotic ways, so near misses will not look like ours”
It is tempting to treat machine-learning failures as a separate species. In practice the near misses that surface in AI operations are ordinary operational slips wearing unfamiliar clothing.
- A retraining job picked up the wrong snapshot of a dataset because two folders had similar names.
- An evaluation notebook compared two model versions on different test sets and nobody noticed until a reviewer asked.
- A configuration change that was meant for staging was sitting in a pull request targeting production.
- A batch scoring job finished with zero rows written, and the success flag still turned green.
Every one of these is familiar from conventional software delivery: wrong artifact, mismatched comparison, misdirected change, silent no-op. That is good news. It means practices proven elsewhere transfer directly, and you do not need a new theory of failure to start learning.
If you want a plain-language refresher on how a vision model arrives at its outputs before you hunt for the weak points around it, the site carries a beginner-level walk through how image classification works. Understanding the normal path makes abnormal paths easier to spot.
Myth 4: “Reporting near misses will make the team look careless”
This myth is cultural rather than technical, and it is the most expensive of the five. If a scare is treated as a confession, people stop reporting scares, and the whole organisation forfeits its least expensive early-warning signal.
The managers who run operations groups can change the incentive in three concrete ways. First, thank the person who reported before asking a single question about cause. Second, keep write-ups blameless and focused on the system: the missing guard, the ambiguous name, the dashboard nobody opens. Third, publish a short digest of what was learned and what was changed. Once people watch a report end in a repair instead of a telling-off, volume rises, and the quality of the data improves with it.
A practical measure of health is the ratio of reported near misses to reported incidents. A healthy team sees many more of the former. If the numbers are nearly equal, people are probably reporting only what they cannot hide.
Myth 5: “A good monitoring stack makes near misses unnecessary”
Monitoring catches what you thought to measure. Near misses reveal what you did not think to measure. A team can have thorough latency, error-rate and drift dashboards and still be surprised by a stale lookup table, because nobody imagined that table could go stale.
Treat each near miss as a request from the system for a new check. After a scare involving, for example, a feature that silently changed units, add a range assertion on that feature. After one involving a misrouted deployment, add a guard on the target environment. Over months, the checks you add come from real observed weaknesses, which makes them far more valuable than checks borrowed from a generic list.
For readers weighing how analytics support decisions more broadly, this piece on how machine learning informs business decisions provides the wider context in which these controls sit. A decision is only as good as the data and model beneath it, and near misses are the cheapest audit of both.
Putting the corrections to work
Dropping the five myths does not require new tooling. It requires a few small commitments that can start this week.
- Create the four-field form and make it reachable in under a minute.
- Schedule a fortnightly, no-blame session of thirty minutes to cluster the entries.
- For each cluster, decide on one automatic check that would have made the catch unnecessary to luck.
- Tell the team what changed, so reporting visibly pays off.
A word on measurement. Resist the temptation to build elaborate scoring for near misses. A simple tally by cause category is enough: data, deployment, evaluation, configuration, tooling. After a quarter, whichever category dominates tells you where to invest. If one pipeline appears in half of the entries, you have found your next engineering project without needing a single customer to be harmed.
An illustration from a scoring pipeline
Consider a team whose nightly scoring job writes predictions to a table consumed by a reporting dashboard. One morning an engineer notices the table has fewer rows than usual. The job had reported success. Investigation shows an upstream export had been truncated, the job read the partial file without complaint, and scored what it found. The dashboard was refreshed an hour later by a scheduled task, so no report went out with incomplete data, only because the engineer arrived early.
Under the myth-driven approach this is shrugged off: nothing broke. Under the corrected approach it becomes three actions. The job gains a check that compares its row total with the prior night’s. The export step gains a completion marker that the scoring job must see before starting. And the near miss is logged with its distance from harm, which was one scheduled refresh. Six weeks later, a similar truncation occurs and the new check halts the job before any scoring happens. That is the return on treating a harmless scare as evidence.
Where to read more on operational reliability
The habit of learning from close calls is well established in reliability engineering circles. For a general-interest look at that field, this website covering site reliability engineering offers a blog-style collection of short pieces on keeping services dependable. Its subject is the discipline behind the idea, and teams running AI systems can take on the mindset and leave the specific toolset behind.
Summary
The five myths share a root: they assume that learning requires pain. It does not. The near misses already sitting in your logs, chat threads and half-remembered mornings describe your real weak points more accurately than any speculative risk register. Count them, cluster them, fix the clusters, and let the quiet scares do the teaching.