Read what follows as pure fiction: nobody real is described, yet each mechanism on the page is documented behaviour. A small team runs inference services and a nightly retraining job for a recommendation feature. Their cluster is growing, their invoice is growing faster, and nobody can say why. That puzzle is the daily business of KubernetesOps applied to a machine-learning workload.
Chapter one: the cluster that looked full
The imaginary team runs ten nodes. Each is a 16-CPU machine that, after system overhead, offers 14.5 CPUs to workloads. Ten of them give 145 allocatable CPUs. The dashboards show the services have asked for 120 CPUs in total, so the scheduler sees about 82.8 percent of the cluster as spoken for. New pods start failing to place, and the instinctive answer is to add an eleventh node.
Before approving that purchase, a team lead asks a different question: how much of that 120 is actually used? The monitoring says peak usage, measured at the 95th percentile, sits around 42 CPUs. That is 35 percent of what was requested. The scheduler, it turns out, does not care. As the Ops Atlas explains in its entry on KubernetesOps, requests work as a contract: pods are placed by what they asked for, not by what they use, and the autoscalers reason from the same numbers.
Chapter two: the contract nobody read
The requests had been copied from an old template years earlier, then inflated whenever a model-serving pod was once evicted. Each padded number felt safe locally. Collectively they made the cluster look full while most of the silicon idled. Machine-learning teams are especially prone to this because memory spikes during model loading make people round up aggressively.
The first practical step was an observation-only pass. The vertical autoscaler can run in a mode that recommends values without changing anything, and the team let it watch for a few weeks, including one retraining cycle. The recommendations became a reviewable list, not an automatic rewrite. That distinction matters: nobody woke up to restarted pods.
- Observation first, changes second.
- Recommendations reviewed by the service owners, not by the platform alone.
- Memory reductions treated gently, because the Atlas notes that shrinking memory in place carries no guarantee.
It also helps to say out loud what a request is not. It is not a prediction, and it is not a limit. It is the amount the scheduler sets aside on a node so the pod can start, and every other decision downstream treats that reservation as fact. When the reservation is wrong, placement, scaling and even the order in which pods are evicted under node pressure all inherit the error. In that last case the Atlas says usage above requests is what gets pods evicted first, so under-asking is risky in a different way from over-asking.
Chapter three: the arithmetic of rightsizing
The Atlas offers a worked heuristic that the team borrowed: take the 95th-percentile usage and add roughly thirty percent headroom, then aim to keep the cluster around a 75 percent target. Applied here, 42 CPUs of real use becomes about 55 CPUs of requests. Divided by the target utilisation, that fits on six nodes rather than ten. The invented team had not been short of capacity at all; they had been short of honest numbers.
There is a guardrail attached to that kind of saving. The cluster autoscaler, by default, only removes a node when it is below fifty percent requested. Before the cleanup, no node ever dropped that low, so nothing was ever released. After the requests came down, the machinery that had been idle for months started doing its job. The change in behaviour came from data, not from a new tool.
Chapter four: autoscaling that finally made sense
With requests corrected, the team revisited horizontal scaling for the inference pods. The scaling formula is simple: desired replicas equal the current count times the ratio of the current metric to the target, rounded up. With four replicas running at 90 percent CPU against a 60 percent target, the answer is six. At 64 percent, the small tolerance band means nothing happens, which is why a service can sit slightly above target for a long time without growing. Scale-down additionally waits for a stabilisation window of about five minutes, so brief lulls do not cause thrash.
One subtle point: CPU-based scaling only works if the pods declare CPU requests, because utilisation is measured against the request. For months the team had pods with no request at all, and the autoscaler quietly did nothing. Another rule from the Atlas is to avoid combining the vertical autoscaler with the horizontal one on the same CPU or memory signal, because they pull against each other.
Chapter five: what the team stopped doing
The retrospective was short and a little embarrassing. They stopped adding nodes in response to scheduling failures without checking usage first. They stopped copying resource blocks between services. And they began tracking two numbers monthly: the share of requested capacity, and the ratio of real usage to requests. The Atlas lists both among its suggested indicators, along with version lag and restore readiness.
If this sounds more like an accounting exercise than machine learning, that is the point. Plenty of teams treat infrastructure as background noise until the bill arrives. The same habit of questioning inputs is what makes analysis that supports business decisions trustworthy in the first place: a model fed padded numbers gives confident nonsense. And if you are earlier in your own journey, the skills roadmap for DevOps engineers is a sensible place to learn where cluster operations fit.
Epilogue: the KubernetesOps checklist the fictional team pinned up
- List every workload and its declared requests; flag any without them.
- Compare requests to 95th-percentile real usage over a full cycle.
- Run recommendations in observation mode and review them with owners.
- Rebuild the capacity maths: allocatable, requested, used.
- Only then decide whether to add or remove nodes.
Nothing in this checklist is exotic. KubernetesOps rarely is. It is mostly the patience to read the contract your cluster is actually running on, and the willingness to renegotiate it when the numbers no longer match reality.