Teams that run model-backed products tend to reach for reliability vocabulary the moment something embarrassing happens in production: a recommendation widget that hangs, an inference endpoint that answers slowly for an hour. The vocabulary they reach for is usually SRE, short for site reliability engineering. Below are the questions an AI-focused group tends to raise on first meeting the discipline, answered with the framing in the Atlas page on SRE from xOps School as the cited reference, with an analytical eye on what the numbers really say.

What is SRE, in one paragraph?

It is the practice of treating operations as a software problem. You decide what experience users should get, express it as a measurable target, and let the gap between target and reality steer how quickly changes ship. The Atlas credits the idea to a small production team assembled at a large search company in the early 2000s, and quotes the one-line summary that it is what happens when you ask a software engineer to design an operations team. The name is more than twenty years old; the habits it describes are what matter.

How is an SRE service level indicator different from a dashboard metric?

A dashboard metric is whatever the system happens to emit. An indicator is a ratio of good events to valid events, taken at the point closest to whoever is using the product. For an AI service this distinction bites hard. GPU utilisation, queue depth and token throughput are interesting, but none of them says whether the person who asked a question got a usable answer in a sensible time.

Latency deserves a special note. The reference material insists on expressing it as a proportion of requests faster than a threshold rather than as an average, because an average hides the slow tail that users remember. If an inference route answers most requests quickly but one in fifty stalls, the mean looks healthy while the experience is not.

Why shouldn’t we aim for 100 percent?

Because it is unreachable and expensive, and because it freezes change. The Atlas puts it bluntly: every added step toward perfection can run to a hundred times the price of the step before. It also offers a “rule of the extra nine”: a dependency should be more reliable by one nine than the service built on top of it, otherwise your promise is capped by something you do not control. A product that calls an external model endpoint should check that arithmetic before it publishes a target.

What is an error budget, numerically?

One minus the objective. Take a 99.9 percent target over 28 days: about 40 minutes of badness, or a thousand failed requests per million. The sheet gives a tidy worked case. Fifty million valid requests with that target yields a budget of fifty thousand failures; if thirty-one thousand fail, the measured success rate is 99.938 percent, the objective is met, and about sixty-two percent of the budget is spent.

Notice how that reframes an argument. Rather than debating whether the last incident “felt bad”, the team asks how much of the budget it consumed. One practical rule from the Atlas: if a single incident eats more than a fifth of the budget, it earns a postmortem with a top-priority action item attached.

How should alerts relate to the budget?

Alert on the speed of spending, not on every blip. This is called burn-rate alerting. For a 99.9 percent target, the workbook example pages a human when the budget is being consumed at 14.4 times the sustainable pace over an hour (confirmed over a five-minute window), and again at six times over six hours. Slower leaks, at three times over a day or equal to the sustainable pace over three days, raise a ticket instead of a page.

Two edge cases from the sheet are worth remembering. At a 90 percent objective, a rule that pages when two percent of the budget disappears in an hour can never fire, since the maths does not allow it. At five nines, a complete outage drains the whole budget in roughly twenty-six seconds, so no human can respond in time and the design must lean on automation. Both cases show that alert thresholds cannot be copied between services without rechecking the arithmetic.

Where does machine learning fit into SRE?

Mostly outside of it. The Atlas scopes correlation and anomaly detection into AIOps, a sibling field, and keeps SRE focused on objectives, budgets, toil, postmortems, on-call load and capacity. That is a useful boundary for an AI-oriented team: clever tooling can help you notice problems, yet deciding what reliability level to promise, and what to do when you miss it, remains a human agreement. For a reminder of how decision quality depends on honest inputs, see this article on how data-driven choices improve business decisions.

What counts as toil, and how much is acceptable?

Toil is work that is manual, repetitive, automatable, tactical, without lasting value and growing in step with the service. Google’s stated ceiling is less than half of an engineer’s time. Quarterly surveys inside the company averaged about a third, with individual engineers anywhere from nothing to eighty percent, and a 2026 industry survey with 418 responses reported a median of 34 percent. A simple audit: six engineers with eighty hours each is 480 hours; if 190 of them go to repetitive chores, that is 39.6 percent, under the cap but not by much.

The cap only works when it has consequences, such as redirecting tickets back to developers or returning developers to the on-call rota. Without that, it is a slogan.

What are the usual ways teams get SRE wrong?

• Renaming an existing operations group and changing nothing else.
• Promising five nines everywhere because it sounds impressive.
• Writing objectives that nobody with authority has signed.
• Alerting on every available metric until the pager is ignored.
• Averaging ratios instead of summing numerators and denominators first, which skews low-traffic services.
• Letting services without instrumentation drop silently out of a query.

Where should a newcomer start?

Pick one user-facing path, write its indicator, set a modest four-week objective, and compute the budget by hand before you automate anything. Anyone still at the starting line may prefer a skills overview first, and the roadmap of skills for aspiring DevOps engineers works as a decent companion. Then run one review after your first notable miss, with every action given an owner and a date. Repeat that loop and you have covered the bulk of the day-to-day practice. A short closing thought for model-heavy products: retraining schedules, prompt changes and data refreshes are all releases, so they belong inside the same budget conversation as any code deploy, and a bad batch of them should slow the next one down in exactly the same way.