Plenty of machine learning initiatives stall, and a quiet culprit sits behind many of them: vocabulary. A data scientist, a product manager and an operations lead can sit in the same review, nod at the same slide, and leave with three different ideas of what was promised. Shared AI terminology sounds like a soft topic. In practice it decides whether a model launch is judged a success or a disaster. Below are six common myths about the words AI teams use, and what to do instead.
Myth 1: “Accuracy” means the same thing to everyone in the room
It almost never does. To the modeller, accuracy may be the share of correct predictions on a held-out test split. To the product owner, it may mean “customers stop complaining”. To a compliance reviewer, it may mean “no wrong answers on the cases that matter legally”. All three are reasonable. None of them is interchangeable.
The plain statistical metric is also easy to misread on imbalanced data: a fraud model that labels every transaction as legitimate can look impressive on paper while catching nothing. Anyone who wants a refresher on the textbook sense of the term can read this site’s primer explaining accuracy with examples and use cases. The bigger lesson is organisational. Before any number goes on a slide, write one sentence stating exactly what “accurate” means for this model, on which population, measured how, and who signs off when that meaning changes.
Myth 2: Labels are objective facts, so nobody needs to define them
Training labels feel like ground truth because they are stored in a column called “label”. Yet every label is a human or rule-based judgement. Is a support ticket “urgent” because the customer said so, because a contract tier says so, or because an agent decided so? Is a product review “negative” if it gives three stars with a polite complaint?
When annotators work from loose guidance, the model learns their disagreements. Teams comparing labelling and annotation platforms often focus on throughput and tooling, but the deciding factor is usually the labelling guide: a written definition of each class, borderline examples on both sides, and a named person who resolves disputes. Without that document, inter-annotator agreement stays low no matter which software you buy.
Myth 3: “Production” is a single, obvious state
Ask five engineers when a model is “in production” and you may hear five answers. Deployed behind a feature flag for internal staff? Serving a small share of real traffic? Fully replacing the old rules engine? Merely registered in a model store with a “prod” tag?
This matters because commitments hang on the word. Monitoring budgets, on-call responsibility, data retention and customer communication often switch on at “production”. If the data science team thinks a model is still experimental while the support team thinks it is live, gaps appear exactly where nobody is watching. A short shared ladder helps, for example:
- Prototype: runs on a laptop or notebook; no real users.
- Shadow: receives real inputs, outputs are logged but never shown to users.
- Limited release: a defined slice of users sees the outputs, with an agreed rollback trigger.
- General availability: default path for all eligible traffic, with a named owning team.
Notice that each rung describes something observable: who sees the output and who answers for it. That is deliberate. A label such as “almost live” or “basically shipped” cannot be checked, so it cannot settle an argument. A rung that says “a defined slice of users sees the outputs” can be verified by anyone with access to the routing configuration, which is exactly what a shared term should allow.
The exact steps can differ by organisation. What matters is that each is written down once and referenced everywhere, from tickets to board updates.
Myth 4: “Drift” is a technical term with a precise meaning
“The model is drifting” is one of the most loosely used sentences in AI operations. Sometimes it means the input distribution shifted: new customer segments, a new device type, a seasonal pattern. Sometimes it means the relationship between inputs and outcomes changed, so the same features now predict something different. Sometimes people simply mean “performance dropped and we are not sure why”.
Each interpretation leads to a different response. Input shift may call for re-weighting or fresh sampling. A changed relationship may require relabelling and retraining. An unexplained drop may be a broken upstream join that has nothing to do with the model at all. Teams that keep a shared glossary entry for drift, with the specific checks that confirm each type, waste far less time debating whose problem it is.
Myth 5: Business stakeholders will adapt to the data team’s vocabulary
Data teams sometimes assume the rest of the company will learn their language. In reality, executives and frontline staff bring their own definitions, rooted in years of reports and habits. Finance has a firm idea of what an “active customer” is; marketing may have another; the churn model may have been trained on a third.
The disciplined approach is to treat business terms as shared assets with an owner, not as informal labels. Several organisations use terminology management tools or a business glossary for this purpose, but a well-maintained wiki page can work if someone is accountable for it. The key fields are simple: the term, its single approved meaning, the source table or calculation, the owner, and the date it was last reviewed.
This is ordinary data governance, and it predates AI. Training and consulting providers such as DataOpsSchool, which helps data teams build quality and governance habits, treat agreed definitions as the base layer that everything else, from testing to observability, depends on.
Myth 6: Once a definition is written, the job is done
Definitions decay. A product launches in a new market and “customer” suddenly needs to cover a reseller. A regulation changes what counts as “personal data”. A retraining cycle introduces a new label class. If nobody owns the glossary, the written meaning drifts away from actual practice and people stop trusting it.
Practical ownership looks like this:
- Each important term has one named owner, usually in the business function that cares most about it.
- Changes are proposed openly, with the reason and the systems affected.
- Downstream users, including model owners, are notified before the change takes effect.
- Old meanings are kept with dates, so historical results can still be interpreted correctly.
This is lightweight compared with the cost of a model being retrained on a silently redefined target.
Where to start: a short vocabulary agreement
You do not need a company-wide programme to benefit. For your next AI initiative, try a one-page vocabulary agreement before any modelling begins:
- List the five to ten words that will appear in success criteria, dashboards or launch notes.
- Write one plain-language meaning for each, plus the exact calculation where a number is involved.
- Add one borderline example per term showing what is in and what is out.
- Name the owner for each term and the person who settles disagreements.
- Store the page somewhere everyone already looks, and link to it from the project brief.
Revisit the page at each milestone. If the meaning of a term had to change, record why. Over time this small document becomes one of the most valuable artefacts the project produces, because it lets future teams understand what past results actually measured.
Why precise language is an AI engineering skill
It is tempting to treat definitions as paperwork that slows down real work. The opposite is closer to the truth. Models are mathematical commitments: they optimise exactly what they are told to optimise, on exactly the data they are given. If the humans around them cannot agree on what the target, the label or the success measure means, no amount of tuning will rescue the project.
Teams that invest an hour in shared terminology up front usually find that reviews get shorter, escalations get calmer and post-launch surprises shrink. The words were always there. Writing them down, and agreeing who owns them, is what turns them from a source of confusion into a foundation the whole team can build on.