Upgrade & Secure Your Future with DevOps, SRE, DevSecOps, MLOps!

We spend hours on Instagram and YouTube and waste money on coffee and fast food, but won’t spend 30 minutes a day learning skills to boost our careers.
Master in DevOps, SRE, DevSecOps & MLOps!

Learn from Guru Rajesh Kumar and double your salary in just one year.

Get Started Now!

Why Your ML Experiments Keep Repeating and How to Fix It

Why Your ML Experiments Keep Repeating and How to Fix It
Artificial Intelligence

Symptom: two engineers spend a fortnight on the same model tweak, neither knowing about the other, and the result is a pair of nearly identical notebooks that disagree on one metric. Nobody did anything wrong. The work simply happened behind closed doors. This diagnosis covers why machine learning teams so often repeat experiments, how that hidden effort costs real time, and the short list of changes that bring the work into view.

The problem: experiments that live on one laptop

Most AI projects start informally. Someone trains a model, tries a few settings, keeps notes in a scratch file, and moves on to the next idea. Months later, the knowledge sits in three places: a laptop, a chat thread, and the memory of whoever ran it. When that person is on leave, the project cannot answer a basic question such as “did we ever try a smaller learning rate on this data?”

The cost shows up in three forms.

  • Duplicate runs. Compute and hours go to repeating a trial that someone already ran and discarded.
  • Quiet failures. A dead end is rarely written up, so the next person walks into it too.
  • Late review. By the time anyone else sees a result, it has hardened into a conclusion, and challenging it feels expensive.

Root cause: progress is only visible at the finish line

The underlying fault is not carelessness. It is a habit of showing work only when it feels finished. For model work, that moment may be weeks away, because training, evaluation and cleanup take time. In the gap, nothing is visible to anyone else, so the team cannot coordinate around what is in flight.

Software engineers met a version of this problem long ago, and the fix they settled on is worth borrowing. Keep the work where the whole team can see it while it is still rough: public issue trackers, open design documents, pull requests that are visible from the first commit. The same tools suit model work with small adjustments.

Fix 1: an experiment log the whole team can read

Start with a shared page or table where every run gets one row before it starts, not after it ends. Minimum fields: the question being asked, the data slice, the main settings, who is running it, and a status. Add the outcome and a one-line takeaway when it finishes, including for failed runs.

Writing the row first matters. It lets a teammate see “someone is already trying that” and either wait, join in, or choose something else. Writing it afterwards only documents a duplicate.

Fix 2: a short design note, posted early

Before a larger piece of work, write half a page: the goal, the approach, what you expect to go wrong. Post it in the open and invite comments for a few days. This is not a formal gate. It is a cheap way to hear “we tried that two quarters ago” while the idea is still easy to change.

For readers newer to the field, an accessible walk-through of the ground being covered, such as our simple guide to how image classification models read pictures, is exactly the sort of background that makes a design note easier for a wider audience to comment on.

Fix 3: visible pull requests for pipeline changes

Training scripts, feature code and evaluation harnesses should go through the same visible review as any other code. Raise the pull request early, label it a draft, and spell out which parts still fail. Reviewers then catch a leaked label or a mismatched split before the run, not after the headline number has already been shared with stakeholders.

Fix 4: put dead ends on the record

Create a heading in the log called “did not work”. Entries can be two lines long. With a few months of entries this turns into the section people open first, since it stops repeats and shows newer teammates how the team thinks. It also lowers the sting of a failed experiment: when failures are routinely posted, nobody has to hide one.

A note on audience and trust

Visibility is not only for engineers. Business colleagues who depend on the model’s output tend to trust a system more when they can see how it evolved. A readable record helps when someone asks why a prediction changed, and it complements pieces like our piece on the way AI supports choices made inside a company, which assumes that the people relying on a model can find out what stands behind it.

The same principle runs through everyday tools far from machine learning. A local-shopping website such as BuyMLocal presents itself as somewhere people search for products and services close to home and see availability put up by nearby shops, which is a reminder that information posted where others can see it gets used far more than information kept to oneself.

Common objections, answered briefly

“It will slow us down.” Writing a row takes a minute. A duplicated week of training takes far longer.

“Our results are not ready to be seen.” That is the point. Rough results are where outside eyes help most, and the log can say plainly that a result is preliminary.

“Some data is sensitive.” Then keep the data private and share the question, settings and outcome. Openness here covers the work, not the raw material. Where an experiment touches regulated or confidential inputs, follow your organisation’s own rules about what can be posted and to whom.

“Nobody will read it.” Early on, perhaps not. Mention the log in stand-ups, link it in tickets, and it becomes the first place people look.

A one-week rollout

  1. Day one: create the log with the five fields and add the runs currently in flight.
  2. Day two: agree that new runs get a row before launch.
  3. Day three: add the “did not work” section and seed it with two or three known dead ends.
  4. Day four: write the next design note and post it for comments.
  5. Day five: open any pending pipeline change as a draft pull request.

Teams moving into this field from operations work often find the habit familiar; a roadmap such as the skills roadmap for new DevOps engineers covers tools that already encourage shared visibility, and the same attitude carries over.

Signs the habit has taken hold

Watch for small events instead of charts. Someone replies “I already have a run on that, here are the settings”. A reviewer flags a data issue in a draft instead of in a post-mortem, and the fix costs an hour rather than a rerun of the whole training job. A new joiner reads the “did not work” list and skips a trap. When those start to happen weekly, the hidden-work problem is shrinking, and the team spends its compute on questions that are actually new.

Subscribe
Notify of
guest
0 Comments
Oldest
Newest Most Voted
0
Would love your thoughts, please comment.x
()
x