Every machine learning team has a handful of small irritations that nobody owns: the notebook that works on one laptop only, the training run that dies at hour six and starts over, the evaluation number that shifts for no clear reason. A practical way to deal with them is to pick one per month and fix it at its cause rather than patching around it again. Below are eight candidates from day-to-day AI engineering, written as a menu you can choose from, together with what a proper fix looks like for each.
How to use this list
Resist the urge to take on all eight together. Choose the item your team complains about most often, give it a single focused effort, and write down what changed. The goal is a lasting removal of friction, so each entry names the usual symptom, the underlying cause and a fix that should hold. Teams working on AI operations in particular tend to find that the cumulative effect of eight small removals is larger than one ambitious tooling project.
1. Notebooks that only run on one machine
The symptom is a colleague saying "it works for me" while you stare at an import error. The cause is almost always an environment that exists only in someone’s head: unpinned package versions, a local data path, a kernel that was configured by hand. The fix is to capture the environment as a pinned dependency file and replace local paths with a single configuration entry. Run the notebook once on a clean machine to prove it. After that, the fix is a one-time cost and the complaint disappears.
2. Training runs that restart from zero
A job that is interrupted by a preempted node or a full disk and then begins again from the first step wastes both compute and morale. The root cause is missing or untested checkpointing. Fixing it means saving model and optimizer state at sensible intervals, confirming that a resume reproduces the same loss curve within noise, and recording the resume procedure where the whole team can find it. Test the resume path deliberately by stopping a small job on purpose; an untested checkpoint is a hope, not a fix.
3. Evaluation numbers that wobble
When a metric moves by a visible amount between two runs of identical code, people stop trusting it. Typical causes are unseeded randomness, a data split that is regenerated on every run, or a test set that quietly changed. The fix is to freeze the split, set seeds in every place that uses randomness and report variation across several seeds rather than a single figure. Once the wobble is understood, genuine regressions become much easier to spot.
4. Slow data loading that starves the accelerator
If utilisation graphs show the expensive hardware idling while the input pipeline catches up, the annoyance is the pipeline, not the model. Profile the loader before changing anything. Common culprits include decoding images on a single worker, reading many tiny files and repeated preprocessing that could be cached. A monthly fix might be as narrow as converting a dataset to a format that reads sequentially and measuring the before-and-after step time.
5. Prompts and configs scattered through code
For teams building on language models, prompt text and numeric settings often end up pasted in several places, each slightly different. Changing behaviour then means hunting through the repository. The fix is to move them into one versioned configuration location that the code reads, so a change is a reviewable diff. Our own overview of how AI feeds into business decisions touches on why traceable inputs matter once outputs start informing real choices.
6. Long waits for shared compute
Queues for a shared cluster breed work-arounds such as hoarding reservations. The underlying issue is usually that jobs are requested with generous guesses for time and memory. Spend the month measuring what your typical jobs actually use, then adjust the defaults and share the numbers with the team. Smaller, accurate requests tend to start sooner, which removes the incentive to hoard.
7. Mystery labels in the training data
Few things are more irritating than discovering that a class means different things in different files. The fix is a short, agreed label definition kept next to the data, plus a check that rejects values outside the agreed set. Even a small vision project, like the one in this beginner-friendly guide to image classification, depends on labels meaning the same thing everywhere.
8. Model files nobody can place
A folder of files named after a date and a mood is an annoyance every time someone needs to roll back. The fix is a small registry convention: each saved model records the code revision, the data snapshot and the evaluation result that justified keeping it. This takes an afternoon and saves far longer the first time a rollback is urgent.
Choosing the month’s item
A simple scoring approach works well. For each candidate, estimate how often it interrupts someone per week, how many people it affects and how long a proper fix would take. Favour the item with the highest interruptions per hour of fixing. If two tie, prefer the one whose fix also makes a second item easier; for example, pinned environments make evaluation wobble easier to diagnose.
A brief aside on browsing for ideas
When teams decide how to lay out an internal list of models and services, some glance at how outside sites do it. BestToolsX, for instance, is a demo-style directory of technology products whose pages say the entries are sample data, and its listings can be found by product type or by the problem they target. Treat that as layout inspiration at most; the fix for each item above lives in your own repository.
Finishing properly
A fix is finished when three things are true. The change is merged, so nobody has to remember it. A short note explains what was wrong and what now prevents it, so the next newcomer does not reintroduce it. And the original complaint has been retired from the team’s informal list of grumbles. If you cannot tick all three, the fix is only half done and the annoyance will return.
After a few months, keep a one-line log of each fix and the symptom it removed. Reviewing that log is a quietly satisfying exercise, and it makes a strong case when you need to justify protected time for maintenance-style improvements.
Closing thought
Small recurring annoyances shape how a team feels about its work far more than their size suggests. Picking one a month, fixing it at the source and recording the result keeps momentum without requiring a grand programme. Start with whichever item above made you wince while reading it.