A model-serving host starts returning answers in four seconds instead of one. The dashboard says processor use is a calm 40%, the model weights have not changed and nobody deployed anything. This is the kind of puzzle where SysOps, the operational care of the underlying Linux machine, earns its keep, because the fault usually lives below the application. Here is a structured diagnosis, from symptom to remedy, for teams running inference or data jobs on their own hosts.
The symptom, stated precisely
Write the complaint as a measurable sentence before touching anything: “requests to the scoring service take about four seconds at the 95th percentile since mid-morning; average processor use reads 40%”. A sentence like that rules out guesswork. It also exposes the first trap, which is that an average is a poor witness. Many monitoring tools sample every five minutes, and the Atlas entry on SysOps lists bursts hidden by those long averages as one of three standard reasons for a slow machine that looks idle. The other two are stolen time on a shared virtual host and saturation of some resource other than the processor.
Step one: do not restart yet
The reflex to bounce the service is strong when users are waiting. The Atlas names restart-first as an anti-pattern because a restart destroys the evidence: caches, queue depths and stuck processes vanish at the moment you most need them. Capture first. A minute spent saving the output of a handful of standard commands is cheaper than a recurrence next week with nothing to compare.
Step two: run a fixed triage list
The Atlas refers to a well-known ten-command checklist from 2015 that takes about sixty seconds and covers load, memory, disks, network and the busiest processes. It also cites the USE method from 2012, which asks three questions for every resource: how utilised is it, is it saturated, and does it report errors? For a model host, the resources worth walking through are the processor, memory, storage and network interface, in that order.
Utilisation tells you how busy a resource is on average. Saturation tells you whether work is queueing for it. Errors tell you whether it is misbehaving. A resource can be lightly utilised and still saturated in short bursts, which is exactly how an inference queue produces slow tails with a quiet average.
Step three: read the signs that mislead
- High load with idle processors. The load figure on Linux includes tasks sleeping uninterruptibly, usually waiting on storage. A burst of model-file reads from a slow volume can raise load without using any processor time.
- Low “free” memory. The page cache fills spare memory on purpose. The figure to read is the amount available to applications, not the amount unused. Weights loaded from disk repeatedly will sit in that cache, which is a feature.
- Disk utilisation near 100%. On solid-state drives and RAID sets the percentage is misleading because the device handles requests in parallel; a reading of 100% does not prove it is out of capacity.
The kernel also exposes pressure-stall information, introduced in 2018, which reports how much wall-clock time tasks lose while queued behind the processor, memory or storage. It answers the question “are tasks actually stalled?” more directly than any utilisation figure, and it is a sensible signal to alert on.
An illustration, kept generic: a team notices that every quarter-hour the tail latency triples. Sampling at one-minute resolution reveals a housekeeping job that compresses old logs at exactly those moments, flooding the same volume that holds the model files. Nothing was wrong with the model or the processor; two workloads simply shared a narrow path. Moving the job to another device and shifting its start time removed the spikes without a single change to the serving code.
Step four: match the evidence to a cause
Suppose the captured numbers show low processor use, rising pressure on input and output, and a growing set of uninterruptible tasks. The likely story is that the host is waiting on storage, perhaps because a log directory ballooned or a batch job started reading the same volume as the model files. If instead steal time is visible on a virtual machine, the hypervisor is giving cycles to neighbours, and no change inside the guest will cure it; the remedy is a different instance or placement.
When memory is the culprit, a different pattern appears: the kernel kills a process to reclaim space. The Atlas warns against the shortcut of setting an extreme score to make a process unkillable, because that merely moves the kill to another victim. Resize the memory, cap the workload or fix the leak.
Step five: fix the cause, then the detection
A fix has two halves. The first removes the immediate cause, for example by moving the batch job to a different volume or schedule. The second makes sure the next occurrence is caught in minutes. The Atlas lists “four signal alerts” in its thirty-day plan, which are alerts on the handful of quantities that indicate trouble on any host. Add finer sampling, perhaps one-minute resolution during business hours, so bursts stop hiding inside averages.
If your team is also exploring how analytics feed decisions, the piece on improving business decisions with machine intelligence explains why the reliability of the input pipeline matters; a stalled host quietly degrades whatever sits on top of it. Likewise, a reader building throughput-sensitive pipelines may enjoy the discussion of visual systems that avoid bottlenecks, which shares the same saturation mindset.
Common mistakes during the diagnosis
- Disabling the host firewall or mandatory access control to rule it out. The Atlas says these controls should stay enforcing, and any exception should be narrow: one domain or one profile, never the entire host.
- Making an unrecorded hand edit while experimenting. Every change should be written down, otherwise the next engineer inherits an undocumented machine.
- Trusting a single dashboard. Cross-check at least two sources before declaring a cause.
- Forgetting file-descriptor limits. A service can stall when it reaches the default soft limit of 1024 open files, even though the hard ceiling is far higher; raise it in the service definition, not globally.
A compact runbook for next time
- State the symptom with a number and a percentile.
- Save evidence before any restart.
- Run the fixed triage list and note utilisation, saturation and errors per resource.
- Check pressure-stall figures and steal time before blaming the application.
- Apply one change at a time and record it.
- Add or refine an alert so the symptom is visible earlier.
The pattern that emerges is that slow-but-idle machines are rarely mysteries; they are queues waiting somewhere nobody measured. A disciplined routine turns an anxious afternoon into a short, repeatable procedure, and it keeps a model-serving host dependable without heroics.