Upgrade & Secure Your Future with DevOps, SRE, DevSecOps, MLOps!

We spend hours on Instagram and YouTube and waste money on coffee and fast food, but won’t spend 30 minutes a day learning skills to boost our careers.
Master in DevOps, SRE, DevSecOps & MLOps!

Learn from Guru Rajesh Kumar and double your salary in just one year.

Get Started Now!

A Night Shift That Learned to Count: A Fictional SOC Ops Story

A Night Shift That Learned to Count: A Fictional SOC Ops Story
Artificial Intelligence

Neither the team nor the company in this story is real. It is a composite sketch, written to show what SOC Ops looks like when a security operations centre stops measuring itself by how busy it feels. Imagine a regional retailer with a six-person monitoring group, a SIEM that collects logs from stores and cloud accounts, and a manager who is tired of hearing that the team is “slammed” without being able to say why.

Chapter one: the month nobody could explain

The manager asked for one thing: a month of dispositions. Every alert a human touched would be closed under one of four headings: genuine threat, harmless real activity, wrong alarm, or still unknown. The labelling was clumsy for a week and routine by the second. At month end the figures were laid on the table.

There had been 4,200 alerts. Only 168 were true positives, a share of 4.2 percent. Harmless real activity accounted for 2,520, wrong alarms for 1,302, and 210 stayed unknown. Nobody in the room was surprised that most alerts were noise. They were surprised by what the noise cost: roughly 72 percent of all triage time went into benign and false items, 424 of 592 analyst hours. Dividing hours by outcomes gave about 212 analyst-minutes for every real incident found.

Those numbers follow the worked month in the Atlas page that frames SOC Ops as a service, and the fictional team’s results are modelled on that illustration rather than measured anywhere. They are useful because they convert a feeling into a budget line.

Chapter two: the benign pile

The biggest bucket surprised them. They had assumed their detections were poor. The tally said otherwise: 2,520 alerts were benign positives, meaning a real thing happened and it was legitimate, such as a vendor scanning an internal range on schedule or an administrator using a tool the rule did not know about. The Atlas points to published research with a similar pattern, where analysts’ “false positives” turned out mostly to be harmless real activity that needed context.

That changes the remedy. A false positive calls for a detection fix. A benign positive calls for context: an allow-list owned by someone, an asset tag, a note on the ticket. The team spent a fortnight adding the context first and rule edits second. Removing a thousand benign alerts, by the Atlas’s own arithmetic, would hand back about 117 hours a month, close to four-fifths of one full-time person.

Chapter three: the night gap

Then came the clock. The group had always reported an overall median time to triage that looked healthy, around seven minutes. An analyst on the new rota thought to split it by shift. After-hours alerts waited a median of more than eight hours. The daytime was fast and the night was a pile that grew until morning.

Averages had hidden it, which is exactly why the Atlas recommends a median and a high percentile per shift and severity instead of a single figure. The team now publishes both on a whiteboard, and the whiteboard embarrassed everyone productively. The manager used the staffing model in the entry, an Erlang-style calculation that turns arrival rate, handling time and target wait into seat-hours, and it said what the team already suspected. Covering a round-the-clock seat takes around five people, not one or two.

Chapter four: the temptation of automation

Someone suggested buying orchestration to close the gap. It is a common move, and the Atlas has a cautionary line on it. Government guidance cited there calls orchestration and response tooling usually unsuitable for immature environments, and says to get the SIEM working properly first. The retailer’s data fit that description: inconsistent log sources, no owner for suppression rules, half-written procedures.

So the team started smaller. A rule could auto-contain a laptop only if it had been proven over time, the asset sat in the lowest tier of importance (ordinary user devices), and a rate limit stopped a bad rule from isolating fifty machines at once. Servers that mattered stayed in human hands. Every suppression got an owner and an expiry date. The rest went into a short list of written procedures for any incident type that arrived about monthly.

Chapter five: what the story is really about

This sort of narrative resonates on a site about machine intelligence because the temptation is always to let a model or a script settle the argument. Our piece on how AI supports business decisions makes the point from the business side: automation is most valuable when a person owns the decision it informs. The same applies when the decision is whether a login at 3 a.m. was an intruder. Nor is prediction a magic fix, as the explainer on how computers guess at market futures reminds readers: an estimate is only as sound as the labelled history behind it. In a SOC that history is the disposition record. No dispositions, nothing to learn from.

  • Count outcomes before changing tools. The month of labels did more than any purchase would have.
  • Treat benign alerts as a context problem. Do not tune away what you have not understood.
  • Split every clock by shift. The night is where averages go to hide.
  • Automate narrowly. Lowest-tier assets, proven rules, rate limits.

Epilogue: the new month

The next month’s tally was not dramatic, and the author of this sketch would be wary of any story where it was. The true-positive share crept up because context removed some noise. The overnight median fell after the manager moved a shift. A handful of undetermined alerts were still undetermined, and the team had stopped pretending otherwise.

That restraint is the lesson. SOC Ops is mostly about being able to say, with evidence, what the team did with every alert and what it cost. If you run a monitoring group of any size, start with the labels. The rest of the programme grows from the record they leave behind.

Subscribe
Notify of
guest
0 Comments
Oldest
Newest Most Voted
0
Would love your thoughts, please comment.x
()
x