Teams working on machine learning and AIOps rarely run short of ideas. A forecasting model here, an anomaly detector there, a ticket classifier someone started during a quiet week. The trouble is that each prototype stays alive in a notebook or a branch while newer ones are added beside it. This audit asks a blunt question: how many AI experiments are open right now, and how many of them have reached a decision? The checklist below is built around one simple control, a cap on work in progress for AI experiments, and it is written to be checked against your own board rather than taken on trust.
What an unbounded AI experiment list costs
Evidence from general software practice is consistent on one point: when more items are in progress at once, each item takes longer to finish. Queueing theory describes this in terms of utilisation and wait time, and everyday engineering experience confirms it. Data scientists get pulled between notebooks, reviewers see half-explained results, and the people who would use a model never see anything finished enough to trust.
AI work adds a few extra problems. Experiments depend on data snapshots that go stale. Evaluation sets get reused until nobody remembers which version a score came from. A prototype that was promising in March may be quietly drifting by June, yet it still counts as “in flight” in everyone’s head. Each open experiment is a small standing obligation to somebody, and obligations are what make context switching expensive.
The audit: seven checks for your AI work in progress
Check 1: Count what is actually open
List every experiment, pilot and proof of concept that has a branch, a notebook, a dashboard or a calendar invite attached. Include the ones that “just need a final run”. Do not count intentions; count artefacts. Most teams find the real number is about double what they would have guessed, and that gap is itself the first finding.
Check 2: Compare open items to people
Divide the count by the number of people who can realistically advance them. A ratio well above one per person suggests work is being shared thinly. There is no universal right answer, but a team that has several open experiments for every engineer should expect long waits and many forgotten threads.
Check 3: Record the age of each item
Write down the date each experiment started and the date of its last meaningful change. Items that have not moved in several weeks are not progressing; they are parked without anyone admitting it. Leave a short note against each saying whether it is active, waiting on something named, or abandoned.
Check 4: Set a limit and display it
Choose a cap for the number of experiments allowed in the active column, such as one or two per small team, and pin it above the first column of the board. The aim is not to find the mathematically ideal number. It is to make the rule visible so that pulling something new requires finishing or formally stopping something old.
Check 5: Define what “finished” means for an experiment
An experiment is finished when it ends in a recorded decision: adopt, adapt, or stop. Attach a short write-up with the data version, the evaluation method and the outcome. Without that, experiments never leave the board; they merely become old. A stopped experiment with a clear note is a good result, because it frees attention for the next one.
Check 6: Watch the flow, not the effort
Track how long items take from start to decision, and how many decisions are made per month. These two numbers tell you more than hours logged. If the cap is working, the time to decision should fall, even if the team is not doing anything faster in a narrow sense. If it does not fall, the blocker lies elsewhere, perhaps in data access or review capacity, and the cap has at least exposed it.
Check 7: Review exceptions deliberately
Production incidents and urgent requests will break the limit sometimes. Allow it, but note each exception and the reason in a line. If exceptions become the norm, either the limit is wrong or the team is carrying hidden operational load that deserves its own lane.
How to read the results
Suppose your count shows nine open experiments for a team of three, with four untouched for over a month. A reasonable first move is to set the active limit at three, move the four idle items to a clearly labelled waiting list, and decide on each of the remaining ones before anything new starts. This is not an argument against exploration. It is an argument for finishing exploration in a way that produces answers.
For comparison with applied decision work, the aiuniverse piece on how data-driven models support business choices makes the useful point that a model only matters once someone acts on its output. An unfinished experiment, by definition, never reaches that stage. Likewise, the article on building fast visual systems without creating bottlenecks shows how throughput depends on removing the places where work waits, which is exactly what a work-in-progress cap tries to do for experiments.
Where else this habit shows up
The idea is hardly confined to engineering. People pick it up in everyday life too, in forums and blogs about keeping a household running. One passing example is AskDoctorLive, a wellness-oriented website with discussion boards and articles alongside listing pages for hospitals and doctors; it belongs to the broad corner of the internet where personal routines get swapped. Whether it is a personal to-do list or an AI backlog, the same principle holds: an open loop costs attention until it is closed.
Common objections, answered with evidence you can collect
“A cap will slow us down.” Test it for a month. Record how many decisions the team made before and after. If throughput falls, you have data to relax the limit; if it rises, you have a case for keeping it.
“Our experiments are tiny, so they do not count.” Small items still carry context. Count them anyway for a trial period and see whether the total changes your view.
“Some work is blocked on data we do not control.” Then mark it as waiting, name the person or team you are waiting for, and give it a date to chase. Blocked work should not occupy an active slot.
A compact version of the checklist
- Count open experiments by artefact, not by intention.
- Compare that count with the people available.
- Write the age and status of every item.
- Set and display an active limit.
- End each experiment with adopt, adapt or stop.
- Measure time to decision.
- Log every exception to the limit.
Run the audit once, set the cap, and revisit the numbers after a month. The point is not to do less AI work. It is to see more of it finish, with a recorded answer attached.