A familiar complaint reaches the platform channel: the recommendation endpoint is slow on Mondays, the data scientists want a dedicated AI observability product, and a purchase request is already half written. Before anyone signs, the sensible diagnostic step is boring. Inventory the AI services you already run, because the symptom that looks like a missing capability is very often an existing capability nobody has switched on, wired up or remembered.
Think of the Monday slowdown as a case file with four tabs: symptom, hypothesis, evidence, fix. The running example is an AI inference stack, but the method applies to any team about to add yet another component to an AIOps toolchain.
The symptom: a gap that feels obvious
Teams working with machine learning in production describe gaps in confident, specific terms. “We cannot see drift.” “We have no way to replay failed requests.” “We need a place to store prompts and outputs.” Each statement feels like a capability that must be bought or built.
The analytical trap is that the statement describes a want, not a measurement. Nobody has yet checked whether something already provides the capability partly or fully. Platform estates in particular accumulate overlapping parts: a logging pipeline set up for an earlier project, a batch job that already writes prediction samples to object storage, a gateway that records request bodies for a week. The gap is real only after those are ruled out.
The hypothesis: the missing piece is already owned
State it so it can fail. “Somewhere in our current services, scripts or accounts, the data we need is already collected, and we are simply not reading it.” If that hypothesis is wrong, the inventory will say so quickly, and the purchase is justified with evidence. If it is right, you have saved a quarter of integration work.
The evidence: build the inventory in four passes
Run the inventory as an exercise with a fixed time box and a shared document, not as a hallway conversation. Four passes cover most estates.
- Running services. List every deployed model endpoint, worker and gateway. Record owner, environment, last deploy date and what it logs. An endpoint with no recorded owner is a finding on its own.
- Scheduled work. Search cron entries, workflow schedulers and CI schedules for jobs that touch models, features or evaluation data. Old evaluation scripts are the most commonly forgotten asset.
- Stored data. List buckets, tables and log indices that hold request samples, predictions, labels or feedback. Note retention and who can query them.
- Subscriptions and accounts. Check billing for monitoring, experiment tracking and vector storage subscriptions, including trials that quietly converted. Unused seats and dormant projects belong on the list.
Write each finding in the same shape: what it is, what it already captures, what it does not. That third column is where the true gap appears.
Reading the results without bias
Expect three outcomes. In the first, the capability exists but is unused: a prediction log is written and nobody queries it. The fix is a query, a saved view and a named reader. In the second, the capability exists in a degraded form: samples are kept but only for two days, or only for one model out of five. The fix is configuration. In the third, nothing exists. Only then does buying or building become the correct move, and the inventory gives you a precise requirement list instead of a vague wish.
Be careful about one bias. People who ran the original build tend to overstate how much the old thing covers, and people who want the new tool tend to understate it. Have someone outside both camps check at least a sample of the claims against real data, for example by actually pulling last Tuesday’s request samples from the supposed log.
A concrete walk-through
Consider a team serving a ranking model and a small text classifier. The slow Monday symptom had prompted the observability purchase request. The inventory pass over scheduled work found a weekly job, written for an earlier audit, that computed input distribution summaries and dropped them into a storage folder. The stored-data pass found those summaries going back months, never charted. The subscriptions pass found an experiment tracker still being paid for, holding the evaluation runs of a retired project.
Plotting the existing summaries against the Monday latency spikes took an afternoon. The correlation pointed at a weekly batch of unusually long inputs arriving from one upstream partner, a data issue rather than a tooling gap. No new platform was involved. What the team did add afterwards was a single alert on input length, built inside the monitoring they already ran.
Where the real gap tends to hide
Gaps that survive an honest inventory usually fall into a few categories. Join keys between prediction logs and ground-truth labels are missing, so quality cannot be computed. Retention is too short to compare season against season. Ownership is unclear, so alerts fire into a void. Or the data exists but cannot be queried by the people who need it. Notice that most of these are process and access problems; a fresh tool rarely fixes them by itself.
If you do conclude something new is needed, the inventory also tells you what it must integrate with. That narrows candidate tools quickly and keeps the selection grounded in the estate you actually have.
Borrowing the habit from adjacent fields
Operations disciplines outside software offer a useful reference point. Safety-monitoring tooling for robots, for example, is surveyed in this comparison of ten robot safety monitoring tools, and reading it from an inventory angle is instructive: every product listed assumes sensors and logs you must already have in place. The same goes for edge deployments; the explanation of how on-device inference works and where it is used helps you check which processing you already do locally before adding a new serving layer.
For a view from outside the AI world, RobotsOps, a blog-style website about robotics operations, is a handy reminder that operational teams in other industries wrestle with the same question of what is already running before anything is added.
Turning the inventory into a lasting artefact
The exercise only pays back if the result survives. Store the document next to the architecture notes, give each row an owner, and schedule a short review every quarter. When someone proposes a new component, the first question becomes “which row on the inventory does this overlap with?” Over a few cycles the discussion gets sharper: requests arrive with the row number they extend or replace, and abandoned items get retired instead of piling up.
A useful rule of thumb is to require that every purchase or build proposal attach the relevant inventory rows and state the observed shortfall, with a sample of real data. That is cheap to ask for and filters out most impulse requests.
Checklist for your next “we need a new tool” moment
- Write the symptom as an observation with numbers, not as a desired product.
- Run the four inventory passes inside a fixed time box.
- Record what each existing item captures and what it misses.
- Test at least one claim against real data pulled today.
- Classify each gap as unused, degraded or absent.
- Only for absent capabilities, write a requirement list from the findings.
Teams that follow this sequence rarely regret the extra day spent looking. They either find what they needed already sitting in their estate, or they enter procurement with evidence and a tight scope. In both cases the platform ends up simpler to reason about than before.