Teams working with artificial intelligence are rarely short of new ideas. A fresh model release, a promising framework or a clever prompt pattern arrives every week, and the roadmap fills up with experiments. What rarely gets a calendar slot is the unglamorous question of whether the AI features you shipped last year still behave. This list gives eight checks for AI already running in production, written for people who would rather prevent a public failure than apologise for one.

Nothing here needs a new platform. Most of it needs an owner, a list and an hour. The goal is simple: before you launch the next thing, find out which of the current things are quietly going stale.

1. Write down what is actually running

Start with an honest inventory. List every model, scoring job, classifier and automated rule that touches real users or real decisions. Include the small ones, such as the keyword filter someone added during a busy week. For each entry note the owner, the data it was trained or tuned on, and where its output goes. You will almost always find at least one item nobody can explain. That item is your first candidate for review.

2. Compare today’s inputs with the inputs it learned from

A model carries a frozen picture of the period when it learned, while live traffic keeps evolving. Customer wording changes, product ranges shift, sensors get replaced. Look at a recent sample of live inputs next to the original training material. If the vocabulary, the formats or the mix of cases look different, performance has probably drifted even if no alarm has fired. A short write-up on how image classifiers learn to recognise pictures is a handy reminder of why a model only knows what it has seen.

3. Re-score a small set of known answers

Keep a modest collection of cases where you already know the right answer, and run it through the live system on a schedule. If the results slip, you have an early signal that is far cheaper than customer complaints. Refresh that collection now and then, otherwise you are only testing the past.

Make this comparison a habit rather than a rescue. Put a calendar entry beside each model and let the person who owns it present a few examples of fresh inputs at the next team meeting. Ten minutes of looking at real traffic often reveals more than a week of staring at an aggregate chart.

4. Check the decisions that people stopped reviewing

Automation begins with a human double-checking every output. Over time the double-checking fades, because the system seems fine. Find the places where review stopped and ask whether it should resume, even on a sample. Writing about AI support for everyday business decisions makes the same point from the business side: the tool informs, and someone still owns the outcome.

5. Walk the dependency chain

An AI feature depends on far more than the model file. There are data feeds, libraries, API keys, vendor endpoints and scheduled jobs. Any of these can expire or change without a word. Trace each one from the feature back to its source, and note the expiry or renewal date where one exists. Credentials that lapse on a weekend are a classic way for a healthy system to stop overnight.

Keep a dated note of each renewal next to the inventory from step one, and name a backup person. Forgotten expiry dates are boring causes of serious outages, and they are entirely preventable with a shared spreadsheet and a reminder.

6. Audit every outbound link in prompts, docs and dashboards

Teams embed references everywhere: in system prompts that cite policy pages, in help text that points to documentation, in dashboards that link to runbooks and in model cards that link to datasets. These addresses rot like any others. A link-management platform such as URLSnow checks the links you publish automatically and flags the ones that stopped responding, which is a gentle way to find out that a prompt is sending users to an error page.

7. Look for a newer option, but only after the old one is understood

It is tempting to replace an ageing model the moment a better one appears. Resist until you can say what the old one does well and where it fails. A swap without that knowledge simply moves unknown behaviour from one place to another. If a replacement does make sense, run both side by side for a while and compare results on your own known-answer set.

Document the reasoning too. A short note saying why you chose to keep, replace or leave a system saves the next engineer from repeating the investigation, and it keeps the decision visible when people ask later.

8. Retire what nobody uses

Every deployed system costs attention, storage and risk. Look for models whose outputs are no longer read, endpoints with no callers and experiments that were never switched off. Turn them off deliberately, archive the code and the notes, and remove the access they hold. A forgotten service with live credentials is a liability that grows with time.

Putting the list to work

You do not have to run all eight in one week. A reasonable pattern is to take two checks per sitting, start with the models closest to customers, and record what you found. When a check turns up a problem, fix it before you resume new work, so the backlog of repairs never grows beyond what you can handle. Pair this with forward-looking reading, such as the primer on how computers produce forecasts about markets, and your team gets both curiosity and discipline.

The hidden benefit is confidence. Once you know the current estate is sound, launching something new feels lighter, because you are not stacking it on an unknown base. Teams that review what they have built tend to ship faster in the long run, since they spend less time firefighting.

It also helps to agree on what counts as a finding. A small wobble in results might just be noise, while a steady slide over several runs deserves action. Decide in advance who gets called, who can switch a feature off, and what the fallback behaviour is, so nobody improvises during an incident. Having a plain, boring answer ready is a sign of a mature team, and it costs very little to prepare.

Pick one item from the list today and put a date next to it. The systems that fail loudly are rarely the ones nobody built. They are the ones everybody built and then forgot.