The story starts with a familiar complaint. A small security group at an imaginary analytics company had a monitoring system that cried wolf all day long, and the manager’s first instinct was to buy something clever: a model that would sort the alerts and quiet the nuisance ones. Six weeks later the model was humming along and the nuisance had barely moved. What they eventually discovered is the heart of DetectionOps, the practice of managing detection rules with the same care a software team gives its code, and it is a useful corrective for anyone in the AI world who assumes the answer to messy signals is always more learning. This is a generic illustration, not a report about a real company.

Symptom: the model could not fix what the rules kept re-creating

The team’s alert stream looked like a leaky roof with a bucket under every drip. Each morning the model re-ranked the pile, and each afternoon the same few rules refilled it. Nobody had asked a simple question: why does this rule fire so often when so little of it is real?

The Ops Atlas page on DetectionOps frames precision in a way that makes the answer countable. Take every alert a rule raised, then divide the genuinely malicious ones by the whole total, counting harmless-but-real activity and plain misfires against the rule. In the Atlas’s worked case, a rule that raised forty alerts had six true findings, thirty benign ones and four outright errors, so only fifteen per cent of its output deserved attention. Sorting those forty alerts more cleverly is a patch; rewriting the rule is the repair.

Diagnosis one: nobody could say why the rule existed

The first pass through the repository, if you can call a shared folder of exported queries a repository, found rules with no stated purpose. Detection logic had arrived over several years from vendor packs, conference talks and one-off requests. The Atlas calls out that last habit directly: importing a big community collection wholesale is a starting point, because even a library of more than three thousand shared rules was written for someone else’s logs.

The fix was dull and effective. For each rule the team wrote a one-paragraph requirement naming the behaviour it hunts for, the data it needs, the known innocent look-alikes and the response expected. The Atlas mentions a strategy template from Palantir with headings for goal, blind spots, false positives, validation and priority, and the group borrowed that skeleton. Writing “blind spots” for each rule forced honest conversations; several rules turned out to exist only to satisfy a checklist.

Diagnosis two: the tests were written from the answer

The team did have a few tests, and they all passed, which felt reassuring until someone noticed they had been generated by replaying the output of the rule itself. That is circular. The Atlas lists exactly this under its anti-patterns: tests written from the rule pass by construction and tell you nothing about the world.

Better tests start from the behaviour. One sample event should resemble the attack and must trigger the rule. Another should resemble the everyday innocent case and must not. When the team accepted a deliberate blind spot, for instance excluding a noisy backup account, they recorded it as a named known-gap test, with someone responsible and a date when the exception lapses. The entry makes the underlying trade-off explicit: every filter buys precision by giving up some visibility, and the cost is only safe when it is written down.

Where machine learning actually fits

Here is the part an AI-minded reader should hold onto. The Atlas places machine-learning and language-model techniques outside the scope of DetectionOps proper; they belong to a neighbouring discipline concerned with applying AI to security data. The two meet in practice, of course. Yet the survey numbers quoted in the entry are sobering. In a 2026 sponsor summary of 307 practitioners, 83 per cent said they use AI tools, but only 42 per cent trust them for tuning detections. The same summary found 59 per cent tracking false-positive rates while only 14 per cent made reducing them a priority.

Those figures come from one sponsor’s survey, so treat them as directional. The story they tell is plausible, though: it is easy to buy a layer that ranks noise and much harder to invest in the unglamorous work of reducing it at the source. If you want the broader context on weighing automated recommendations, our article on how AI supports everyday business decisions covers the habit of checking a recommendation against evidence before acting on it, and the same instinct applies to a model that proposes closing alerts.

Diagnosis three: the pipeline broke itself

Midway through the project the team set up automated checks that linted every rule on each change. For two weeks everything was green, then one Monday every build failed. A tooling library had released a new major version overnight, and the command-line tools that depended on it still required the older generation. The Atlas describes precisely this situation with Sigma tooling in October 2026: pySigma 2.0 had appeared, while the Sigma command-line tool and the Splunk backend still demanded a version below two. The advice is blunt: pin every component of the toolchain so that an upstream release cannot quietly stop your checks.

The team added a lock file and a monthly, deliberate upgrade slot. Boring, yes, but the failure never recurred.

Diagnosis four: fixes made in the console vanished

Another habit nearly undid the whole effort. An engineer, under pressure, edited a rule directly in the monitoring console to silence a flood. The edit worked for a day, then the next automated deployment overwrote it and the flood returned. The Atlas lists console hot-fixes erased by the next deploy among the standard failure modes. Its remedy is a change taxonomy: standard changes follow the full path, urgent changes during an active campaign may skip the staging trial but never testing or review and get re-reviewed within five working days, and filter changes need evidence from analysts plus an expiry date.

Once the team accepted that every edit goes through the repository, even the urgent ones, the overwrite problem disappeared. Fast could still be fast, just recorded.

What the numbers looked like afterwards

Following the Atlas’s example metrics, the team tracked four small numbers each month: the share of rules with tests, the share of high-priority detections validated by emulation within a rolling window, the share of rules in good health, and how many deployments needed rollback. They also borrowed the human-scale unit of analyst minutes per true finding, which made the cost of a noisy rule vivid. The Atlas’s worked example puts one such rule at roughly forty minutes each, a figure that tends to win budget arguments quickly.

The imagined team’s model stayed in place, now doing a modest job of ranking a far shorter list. For the question of whether prediction tools earn their keep in a pipeline, the explainer on how computers make forecasts about markets offers a plain primer on what such systems can and cannot promise.

Takeaways for the next noisy week

• Measure precision per rule before reaching for a ranking layer.
• Give each rule a written requirement and a named blind-spot list.
• Write tests from behaviour, not from the rule’s output.
• Pin the tooling and put emergency changes through the same repository.
• Keep the model for the job it is good at, and verify it before trusting it with tuning.