Pull up a chair, because I want to talk about a handful of beliefs that keep tripping up people who build AI features for a living. Every one of them sounds sensible over coffee. Every one of them has cost somebody a weekend. They all live in the territory called MLOps, the work of getting a trained model into real use and then keeping it honest, and they hang around because they each contain a grain of truth.
I am drawing most of the facts from the MLOps page of xOps School’s Ops Atlas, which is a careful, sourced summary rather than a sales pitch. Let’s take the myths one at a time, friendly tone, no jargon without a translation.
Myth one: “If it scores well in my notebook, I’m basically finished”
The notebook is where a model is born, so it feels like the finish line. Think of it more as a first draft. The Atlas offers a mental model I find handy: treat the model as something you build, where the raw material includes the data. If you accept that, then “finished” has to mean you can rebuild it on demand, and a notebook that was run once on someone’s laptop cannot promise that.
What does a rebuildable run look like? You note which code revision you used, which exact slice of data, which settings and which software environment. Then someone else repeats the run and lands within an agreed margin of your numbers. Note the word margin. Training can involve randomness, and the framework behind a popular deep-learning library openly says results may differ between releases. So “reproducible” in MLOps means close enough to trust, not identical to the last decimal.
Myth two: “A drift dashboard tells me when to retrain”
This is the one I hear most, and I used to believe it too. The picture is lovely: a line wiggles upward, an alarm fires, you retrain, the line settles. The catch is that a single wiggling line cannot tell you why it moved. The Atlas separates three very different causes:
- Skew, where the training pipeline and the live pipeline disagree about how an input is prepared. The cure is fixing the pipeline. Retraining just teaches the model to live with a bug.
- An upstream defect, for example a column that suddenly arrives empty. The cure is at the source.
- Real drift, where the world changed. Even then, the response depends on what kind: the inputs changed, the base rate of outcomes changed, or the relationship between the two changed. Each calls for a different move, from retraining to recalibrating thresholds to rethinking the features altogether.
So the dashboard is a smoke alarm, not a diagnosis. Somebody still has to walk to the room and look.
Myth three: “When a new version misbehaves, just retrain it”
Picture it. A fresh model goes live, quality dips, and the instinct is to throw more training at it. Resist. The reference is firm that you go back by moving a label or a slice of traffic to the previous good version, not by running a new training job under pressure. Training takes time, may not land on the same result twice, and might make things worse while customers wait.
The grown-up version has two parts. First, previous versions are kept and addressable, so “go back” is a configuration change measured in seconds. Second, there is a no-model backup path, for instance a hand-written rule, so that if every model version is suspect the product still does something sensible. The Atlas adds that this should be rehearsed on a regular rhythm, because a rollback nobody has tried is a rumour.
Myth four: “Better model metrics always mean a better product”
Here the Atlas points to a case from the travel-booking company Booking.com, whose engineers reported that gains in model metrics did not automatically become gains for the business. If you have ever celebrated a higher accuracy number and then watched conversion stay flat, you know the feeling.
It is the reason for progressive delivery: show the new model to the world in stages. First it runs in shadow, making predictions that nobody sees while the old model still answers. Then a small canary slice of real traffic. Then an online experiment that measures the thing the business actually cares about. Each step is a chance to discover that a lovely offline score does not translate. A related habit from the same source is gating a new version on the same frozen test set, checking not just the overall number but also slices of users and the response time, so a gain in one place cannot hide a loss in another.
Myth five: “The default alert threshold is the truth”
Tools come with sensible-looking defaults, and it is tempting to treat them like laws of nature. The Atlas is gently skeptical. It notes that the popular bands used with a drift score called PSI, 0.1 and 0.25, lack academic support, and that the score itself shifts with how you slice the data into bins and how big your samples are. It also points out that one open-source monitoring tool, Evidently, declares the whole dataset drifted only once over fifty percent of its columns have moved, a rule that can hide trouble in the single column that matters most.
Practical advice that follows: calibrate alerts on your own history, name the handful of critical inputs and watch them individually, and compare like with like. A holiday-season window set against an ordinary month will always look dramatic. Treat any threshold as a starting guess you tune, not a verdict handed down.
Myth six: “MLOps and AIOps are the same thing with the letters swapped”
On a site like this one the mix-up is understandable, since the two names look like twins. They point in opposite directions. AIOps applies machine learning to the stream of events inside IT operations, sorting alerts and spotting incidents. MLOps is the operations work done for machine-learning systems, the pipelines, registries and monitoring that keep a model healthy. The Atlas also separates it from data engineering and from data science itself: MLOps owns things like training snapshots and the checks for skew, but it is not the job of inventing models.
If you want a taste of the neighbouring ideas, the site has a gentle read on how AI feeds everyday business decisions, and another that walks through the way image classifiers read a picture. Both are the kind of model you would eventually need to operate this way.
One small tale of a currency symbol
Picture a small team running a recommendation widget on a shopping app. They build it, it scores nicely, and they ship it. Three weeks later the clicks sag. One engineer wants to retrain at once. Another suggests checking whether the live site still formats prices the way the training file did. It turns out a currency symbol had started arriving inside the number. That was skew, not drift, and retraining would have baked the bug in. A ten-minute look at the logged inputs saved a day of compute and a week of confusion. None of the six myths above is silly; each is a shortcut that works until the day it does not.
What to do with all this
If three of those myths sounded like your team, here is a small, friendly starting plan. Pick one model that matters. Write down how to rebuild it. Decide, ahead of time, what you will check before a new version replaces it, and where the old version will wait if you need it back. Then set up a monitor on three levels: is the service up, does the incoming data look familiar, and are the answers still good, remembering that real outcomes may arrive late and you need a stand-in measure while you wait.
The Atlas also recalls the opening advice in Google’s ML rulebook, which says a team should hold off on shipping a learned model when a plain approach would do the job. How much to automate is a decision you make, never a setting you inherit. Sometimes the right answer to “should this model retrain itself every night and promote itself automatically” is a polite no, because feedback loops can turn a small error into an outage.
That is the tour. Six comfortable beliefs, six gentler replacements. If you take only one home, make it this: diagnose before you retrain.