Priya had a classifier that everyone liked. It sorted incoming product photos into categories with an accuracy her team was proud of, and for three weeks nobody touched it. Then a colleague retrained it with a handful of new images, the numbers slipped, and the question nobody could answer was simple: slipped compared with what? The old weights had been overwritten. Her story is the clearest argument I know for model version history, and it is worth walking through as a diagnosis, symptom first and remedy last.
The symptom: a better idea that behaves worse
The complaint arrives in many shapes. A recommendation list feels stale. A forecast that used to be steady now jumps around. A vision model confuses two categories it handled yesterday. What these have in common is a before and an after, with no clean record connecting them.
Teams in this position tend to start by hunting for a bug in the code. Sometimes there is one. Far more often the culprit is a quiet change in the inputs: a refreshed dataset, a different preprocessing step, a tweaked learning rate, a library upgrade. Each looks harmless alone. Together they make a mystery, because nobody wrote down which one landed when.
The diagnosis: the work was saved, but the story was not
Priya’s team did store files. They had a folder of checkpoints with names like “final”, “final2” and “use-this-one”. What they lacked was the story that sits beside each checkpoint. Three missing pieces explained nearly all of their confusion:
- Which data produced this model. The training set had been refreshed twice, and the refresh dates were nowhere.
- Which settings were used. Hyperparameters lived in a notebook that had since been edited.
- What the evaluation looked like then. The scores were remembered, loosely, rather than recorded.
Without those, comparing today’s model with last month’s was guesswork. The file existed; the evidence did not.
The remedy: dated versions with a one-line note
The fix was almost embarrassingly small. For every model worth keeping, Priya’s team now saves a numbered version, stamped with the date, and adds a single sentence in plain words: “retrained on the refreshed photo set, added a rotation step, accuracy on the held-out batch rose slightly but fell on dark images”. That one sentence carries the cause, the change and the trade-off.
Alongside it sit pointers to the exact dataset snapshot and the configuration used. Together these make a version something you can reproduce, not merely something you can load.
Comparing versions like a detective
Once history exists, diagnosis becomes methodical. Place two adjacent versions side by side and ask what differs. If the dataset changed and the settings did not, you suspect the data. If only the settings changed, you suspect the tuning. If both changed, you split the difference: try the new data with the old settings and the other way round.
That is the same bisecting habit engineers use on code, applied to models. It only works when every step has a clear predecessor. For readers new to this area, our simple guide to how image classifiers learn to recognise pictures gives the background that makes comparisons like this easier to read, since you need to know what a mistake looks like before you can notice a new kind.
Rolling back without drama
Here the story improves. When the retrained model scored worse on dark images, Priya did not argue about it. She loaded the previous version, pointed the service at it, and wrote a new entry: “reverted to the earlier version while we collect more low-light examples”. Nothing was lost, no one was blamed, and the failed experiment stayed on record with its reasons.
A rollback is only calm when the earlier version is intact and clearly labelled. If your previous model lives in a folder that gets cleaned every Friday, you do not have a history; you have a hope.
Operations teams feel it first
In AIOps work the pain shows up as pager noise. A detection model that flags unusual latency gets updated, and suddenly the on-call engineer is woken twice as often. Was it the new training window, the changed threshold, or a shift in traffic? If the model history carries dates and notes, the on-call lead can line the change up against the first noisy night and answer in minutes. If it does not, the conversation drifts into opinion, and opinions are a poor substitute for a timestamp.
It also helps newcomers. A person joining the team can read the trail from the oldest entry to the newest and absorb, in one sitting, the mistakes the group has already paid for. No wiki page captures that as honestly as a list of small, dated decisions.
What to keep, and what to let go
Nobody wants every experiment kept forever. A sensible rule is to keep each version that was deployed, each version that set a new best score, and each version whose failure taught you something. Short-lived scratch runs can go, provided their lessons made it into a note. Storage is cheap; lost explanations are not.
The same approach applies to decision-support systems more broadly. Think of a forecast that guides a business choice: if you revise the method, record the date and reason, so that when results look odd you can tell whether the world changed or your method did. Our piece on how forecasting models arrive at their market guesses shows how many moving parts a single prediction hides, which is exactly why each revision deserves its own dated entry.
Where instruction text fits in
Teams also accumulate small pieces of reusable text, such as the standard wording that frames an evaluation or a labelling guide. People who like to keep that wording organised sometimes use Promptosia, where reusable prompt instructions can be saved, versioned and shared. Whatever you choose, the principle holds: reusable text deserves the same dated trail as the models around it.
A short checklist to adopt
- Give every kept model a number and a date in its name.
- Write one sentence on what changed and why.
- Record the dataset snapshot and settings next to it.
- Store the evaluation result at the time, not from memory.
- Never overwrite a deployed version; add a new one.
Priya’s classifier is steady again, and, more usefully, her team can now explain its behaviour. That is the quiet payoff of a version trail. You stop asking who broke it and start asking which change helped, and that is a far better question to be arguing about on a Thursday afternoon.