Teams that run machine learning services tend to inherit a monitoring stack long before they decide who looks after it. Then a model endpoint slows down at 2 a.m., the dashboards disagree, and someone says the quiet part out loud: nobody actually runs the thing that watches the things. ObservabilityOps is a name for that missing job, and around it a cloud of half-truths has grown. Six of them follow, each dismantled in turn.
First, what we mean by ObservabilityOps
The label is not an industry standard, and the people behind the Ops Atlas page on ObservabilityOps say as much. They use it for one specific job: operating the observability system itself, meaning the collectors, pipelines, storage back ends, alert rules and dashboards, instead of merely consulting them during an outage. Keep that definition in your pocket; every myth below gets blurry the moment it slips.
Myth 1: “The AI will find the problem for us”
This is the most tempting one on a site about artificial intelligence. Automated correlation and anomaly detection can help, but they sit downstream of telemetry. If the signals are missing, mislabelled or dropped by a misconfigured pipeline, a clever model has nothing honest to chew on. The Atlas lists “AI will find it” among its anti-patterns, and it regards sales claims for machine-driven observability as brochure copy until they have survived a trial on your own data.
The practical test is cheap. Take one past incident, replay the data your tooling had at the time, and see whether the promised feature would have surfaced the cause without you pointing at it. If it would not, the gap is in the plumbing, and plumbing is what ObservabilityOps looks after.
If you want the broader backdrop on where models genuinely help decisions, a piece like this look at how AI supports everyday business choices sets expectations sensibly: assistance, not oracles.
Myth 2: “Observability is just monitoring with a better logo”
Monitoring checks things you thought of in advance. OpenTelemetry’s wording is that observability lets you reason about what is going on inside a system purely from what it emits, and DORA places the boundary in much the same spot, noting that the properties you want to inspect are not fixed beforehand. In practice the difference shows up when something new breaks: a dashboard of predefined charts tells you whether a known thing misbehaved, while rich, queryable telemetry lets you ask a question you had not planned.
That is why the discipline cares about instrumentation standards and consistent fields. A prediction service that logs a trace identifier on every line can be followed through a request. One that does not leaves you grepping by timestamp and hope.
Myth 3: “Metrics, logs and traces mean three separate tools”
You will still hear the “three pillars” slogan, and plenty of buying decisions have been made on it. The CNCF observability working group’s 2023 whitepaper, as the Atlas summarises it, prefers the phrase “primary signals” over pillars, partly to stop the framing from implying three silos. Practically, what matters is that signals can be joined: a log line carries the trace and span identifiers as top-level fields, a metric exemplar points to a trace, and a profile ties to the same service name.
Buy or build for correlation, not for category count. If your stack has three consoles that cannot link to one another, you have three tools and no observability.
Myth 4: “Installing the tracing SDK means tracing works”
An SDK in the dependency list proves very little. Context propagation is where traces actually break: at message queues unless span links are used, at proxies that strip the traceparent header, and in estates that mix two header formats. The Atlas says the W3C trace-context and baggage defaults are the safe baseline, and that baggage can leak to third-party APIs unless you permit only named keys and remove them before requests leave your network.
A model-serving platform makes this vivid. A request goes from gateway to feature service to model server to a queue to a post-processing job. If any hop forgets the header, you get four orphan fragments, not one trace. The repair is a checklist item for the owner of the platform: verify propagation end to end on a schedule, and start a fresh trace at the public edge.
Myth 5: “Collect everything now, decide later”
It sounds cautious, even responsible. Storage is cheap, the story goes, and you never know which log line will matter. The bill disagrees. Every unique combination of label values creates another time series, and one careless label can multiply the count enormously. The Atlas gives a rule of thumb of keeping metric label cardinality to roughly ten or fewer and investigating past a hundred, and it describes a lab example where adding a customer identifier with 50,000 distinct values to one order counter inflated a single metric to 1.8 million series.
The fix is a budget and a review. Teams that run feature stores and inference gateways often have one team producing the bulk of the volume; setting per-team series limits, with a clear message when a limit is hit, is gentler than discovering the issue on an invoice.
- Never place unbounded values such as user or request identifiers in metric labels.
- Put identifiers in traces and logs, where they belong.
- Set a series budget per team and review it monthly.
Before the last myth, a quick aside on timing: none of these corrections needs a big-bang project. Most can be started in a week, by one person, with a spreadsheet and the telemetry you already have. The trick is to pick one myth per sprint and retire it properly, then check again after a month that the habit stuck.
Myth 6: “Running our own stack is free”
Open-source software is free to download and expensive to operate. Disks fill, retention has to be tiered, upgrades arrive monthly, and a stateful back end needs somebody on call. The Atlas also flags a licensing wrinkle worth knowing about: a number of widely used visualisation and storage projects moved from Apache 2.0 to AGPLv3 a few years ago, which matters to some legal teams even if it never touches your code. Self-hosting can still be the right choice, particularly when data cannot leave your network, but price it with people time included.
One handy calculation the Atlas shows: unit cost. Take the monthly bill for the observability stack and divide it by request volume. The example in the entry is a bill of 18,000 over 900 million requests, which is 20 per million. If the figure climbs while traffic is flat, something is leaking.
What a sensible team does instead
Replace each myth with one concrete habit.
- Test any AI-powered feature against a replayed incident before paying for it.
- Write down which questions your instrumentation must answer without a redeploy.
- Require shared identifiers across signals, whatever the tools.
- Test trace propagation across every hop on a schedule.
- Budget series and retention per team.
- Track unit cost next to coverage.
For the model-heavy workloads this site writes about, there is a nice benefit: a stable, well-labelled telemetry layer makes it far easier to compare model versions in production, since the baseline noise is understood. A walk through a primer on how image classifiers read pictures is a decent reminder of how much hidden machinery sits behind every prediction your dashboards are summarising.
A quick sanity check for your own stack
Run this as a fifteen-minute exercise with whoever carries the pager for your inference services, and keep the answers in a shared note so that the next review can compare.
Ask three questions in your next review. Who owns the collectors and the alert rules by name? What happens to your alerts if the alerting system itself goes quiet? And how many of last quarter’s pages led to a real action? If any answer is “we do not know”, you have found your first ObservabilityOps task.
None of this requires a new product. It requires a person, a short inventory and a habit of testing the watchers. The myths survive because they promise to skip those chores. The chores are the work.