Machine learning projects have a reputation for being “almost ready” for a very long time. A model improves a little each week, someone suggests another feature, another dataset arrives, and the launch date drifts. The cure is not a bigger team or a fancier tool. It is a short written statement of acceptance criteria that says what must be true before you call the work complete. This primer explains the idea for newcomers to AI and AIOps and closes with a pocket glossary for the terms you will keep meeting.
What acceptance criteria mean in an AI project
In ordinary software, acceptance criteria describe observable conditions that a ticket must meet: a button saves a record, a page loads within a stated time, an error message appears for invalid input. In AI work the same notion applies, but the conditions are stated in terms of behaviour on data rather than behaviour of a screen. You are describing what the system should do, measured on examples it has not seen, and under what circumstances you would stop improving it.
The reason this matters is that a model can always be made slightly better. Unlike a form with a fixed set of fields, there is no natural last step. Without an explicit finish line, work swells to fill whatever time exists, and the question “are we done?” turns into a debate about feelings.
The anatomy of a good finish line
A useful definition of done for an AI task has a few parts. Not every task needs all five, yet you should be able to point at each and say whether it applies.
- The task, in one sentence. What is the model asked to do, and for whom?
- The measure. Which quantity tells you whether it works, and on which set of examples?
- The threshold. What value of that measure counts as good enough to ship?
- The boundaries. Which inputs are out of scope, so nobody expects the model to handle them?
- The operating conditions. How quickly must it respond, and what must be monitored after launch?
Write these down as plain statements. For example: “The classifier routes incoming support tickets into a small set of queues; on a held-back sample of past tickets it assigns the correct queue at or above the agreed rate; tickets in languages we do not support are sent to a human; responses arrive fast enough that the agent screen does not stall.” That paragraph is a finish line. Everyone reading it can tell when it has been reached.
A concrete example from operations
Consider an AIOps task that many teams attempt early: grouping related infrastructure alerts into a single incident so engineers are not buried. A vague goal might be “reduce noise”. A finish line is sharper. It might read: related alerts from the same service within a short window are merged into one incident; a human can see which alerts were merged and can split them; on a review sample drawn from last month’s alerts, the merged incidents match what responders would have grouped by hand to an agreed degree; and the grouping runs inside the existing alert pipeline without adding noticeable delay.
Look at what that definition does. It names a visible behaviour, a reviewable sample, a human override and a performance limit. It does not promise perfection, and it does not hide the reviewer’s role. When the sample review meets the agreed degree, the work is accepted and further tuning becomes a new, separately scoped task.
Common ways teams get this wrong
Beginner projects trip over the same three mistakes.
The first is choosing a measure that flatters the model. If most of your examples belong to one class, a model that always guesses that class can look impressive. Choose a measure and a sample that reflect the real mix of cases you will meet, and say so in the definition.
The second is leaving the evaluation data undefined. If the examples used to judge the model change from week to week, a rising score may reflect easier examples rather than a better model. Fix the sample, or fix the rule for drawing it, before you start.
The third is treating the launch as the finish line. For AI systems, shipping is a milestone, not an ending, because the world changes and a model can slowly lose accuracy. A complete definition therefore includes what you will watch afterwards and what observation would prompt a review. That is still a finite statement. It simply points a little past the release.
Where this fits among related ideas
If the terminology of machine learning is new to you, a plain overview such as this site’s overview of core machine learning concepts for business teams is a sensible companion read before you start writing criteria. And if your work leans toward running models in production, the career-oriented material on machine learning deployment and operations certification shows how much of the job is about repeatable, checkable steps.
The habit of stating completion conditions also appears in places outside engineering. A blog-style website such as WarOpsX, whose posts are themed around operations, is a reminder that “operations” as a word covers many kinds of organised work, and each kind benefits from knowing where a task ends.
Mini-glossary for beginners
These short entries cover the vocabulary you will meet when writing acceptance criteria. They are simplified on purpose.
- Model: a program that has learned a pattern from examples and applies it to new input.
- Training data: the examples a model learns from.
- Held-out set: examples kept aside and never used for learning, so they can fairly test the model.
- Metric: a number that summarises how well a model performs on a task.
- Threshold: the agreed value of a metric that separates acceptable from not yet acceptable.
- Baseline: a simple reference, such as a rule or a previous system, that the new model should beat.
- Drift: a gradual change in incoming inputs or in the surrounding world that erodes a model’s accuracy as months pass.
- Inference: the act of a trained model producing an output for a new input.
- Latency: the time between a request and the model’s answer.
- Human in the loop: a design where a person reviews or overrides model output.
- Out of scope: inputs or situations the system is explicitly not expected to handle.
How to write your first set of criteria
Follow this short sequence the next time you are handed an AI task, however small.
- Write one sentence describing what the system should do and who benefits.
- Pick one measure and one fixed sample on which you will judge it.
- Agree a threshold with the person who requested the work, in writing, before building anything.
- List what is out of scope, including inputs that should be passed to a human.
- State one or two operating limits, such as speed, and what you will monitor after release.
- Add a line saying what happens when the threshold is met: the task is closed and any extra ideas go into a new ticket.
Step six deserves attention. It is the sentence that converts stopping from an accident into a decision. Without it, a model that meets its target tends to attract new requests inside the same ticket, and the original finish line quietly moves.
Keeping criteria honest
Criteria are only useful if they stay fixed while the work is in progress. If you discover that the threshold was unrealistic, change it openly, record the reason and get agreement, rather than adjusting it silently to match what the model achieved. A moved goalpost is worse than none, because it produces a record that looks like success.
Brevity helps too: aim for criteria you can read through in under sixty seconds. A page of conditions that nobody reads to the end has stopped being a finish line. If a definition grows long, that is often a sign that the task contains several tasks, and each deserves its own sentence of done.
One more habit is worth adopting. When you present the criteria to the person who asked for the work, read them aloud and ask a single question: if every line were true tomorrow, would you be satisfied? A yes means you have a finish line. A hesitation means something important is missing, and it is far cheaper to find that out before the model is trained than after.
Takeaways
Acceptance criteria give AI work an edge that the technology itself will never supply. State the task, the measure, the threshold, the boundaries and the operating conditions. Agree them before you build, keep them stable, and treat meeting them as the moment to stop and start something new. Teams that adopt this habit find that their projects ship more predictably and that arguments about quality become checks against a shared statement. For a beginner, that is one of the most practical skills to learn early.