Every few months a new demo shows a model spotting an intruder in a flood of log lines, and the room goes quiet for a moment. What the demo rarely shows is the unglamorous plumbing that decided which log lines the model was allowed to see. That plumbing has a name among security engineers, SecDataOps, and during 2026 it has been quietly becoming the deciding factor in whether machine-assisted defence works at all. Consider this a progress report for readers who follow AI and automation and want to see what sits beneath the headlines.
A late-night story about missing evidence
Picture a small platform team, entirely made up for illustration. Late on a Thursday an analyst asks the team's anomaly-scoring service why a service account looked odd three weeks earlier. The service answers confidently, yet the analyst cannot find the underlying records. The collection agent had been running on in-memory buffers, a restart wiped a queue, and nobody had written down which sources were supposed to arrive. The model did not fail. The evidence never reached it.
That small drama repeats across the industry. The Ops Atlas has an entry on SecDataOps as the supply chain for defender evidence, and it frames the problem plainly: detections, investigations and audits all need evidence that is complete, trustworthy and affordable. Smart analysis on top of thin data produces confident nonsense.
Where the field stands today
The Atlas is frank that the label is an emerging, vendor-led one. It has been in public use since early 2023, a podcast now carries its name, and analysts have started filing the tooling under a category called data pipeline management. No standards body owns the term. Meanwhile the underlying chores, collecting, parsing, enriching, routing and retaining logs, are older than the buzzword and already described in official guidance from several governments.
For anyone who tracks the AI space, the interesting question is not whether the term survives but what the practice changes about model quality. Three shifts stand out.
Shift one: schemas are converging
For years each platform spoke its own dialect. Microsoft's Sentinel has one model, Splunk another, Google's security platform a third. The Atlas notes that the Open Cybersecurity Schema Framework reached release 1.9.0 in August 2026 under the Linux Foundation, while the Elastic Common Schema continues to converge with OpenTelemetry. A model trained or prompted against one consistent vocabulary needs far less per-source translation, which means fewer silent misreadings.
Shift two: storage tiers are getting cheaper and deeper
Cloud vendors now sell low-cost lakes and tiered plans: Amazon Security Lake stores in a columnar format using the open schema, Microsoft advertises a long-retention data lake, and Azure Monitor offers basic and auxiliary plans where the auxiliary one trades speed and alerting for a minimal ingest price. Cheaper depth matters for machine learning because it allows long baselines. A behavioural model that can only look back ninety days will mistake a quarterly pattern for an anomaly.
Shift three: health of the pipe is now a first-class signal
The older assumption was that if no alert fired, nothing was wrong. The newer one is that silence is itself data. Practitioners track freshness, parse rate and completeness per source, and they page a human when a feed goes dark. The Atlas gives a sample contract with freshness inside 300 seconds at the 95th percentile and a parse rate of at least 99.5 percent. A scoring model fed by a stalled feed cannot warn you that it is blind.
Why dwell time reshapes the retention conversation
Our readers who build predictive systems will recognise the training-window problem. The Atlas cites Mandiant's 2026 report on 2025 investigations, which reports a global median dwell time of 14 days but a far longer 122 days for espionage-style and some insider-style cases, with one malware family lingering roughly 400 days. The same passage quotes the report saying that standard ninety-day log retention leaves organisations blind to the initial access vector.
If your detection depends on learning what normal looked like, short retention quietly shrinks your definition of normal. A model fitted to ninety days and a threat that has been inside for four hundred are playing different games.
Where machines help and where they mislead
It is tempting to assume smarter analytics will rescue messy collection. The Atlas pushes the other way, with a handful of cautionary details that read like bug reports.
- A strict parser in a popular open-source collector turns a null into zero when converting to an integer, so garbage can appear as a connection denied to port 0. The suggested fix is an anchored pattern that quarantines bad lines.
- In-memory buffers in that same tool are not durable, so a restart can drop events; disk buffers are the recommended alternative.
- Syslog cannot acknowledge delivery, and replays double-count unless events are de-duplicated on a stable identifier.
- Some warehouse access logs lag by up to three hours and omit failed queries, which matters if a model learns from them.
Each of these is the kind of flaw that a clever model will faithfully learn from and then amplify. Data quality contracts exist precisely to stop that.
The cost story behind the curtain
Money drives most of the real decisions. The Atlas works through an example of a 400 GB per day firewall stream. Kept entirely in the expensive search tier it costs about 400 units a day, while summaries of 8 GB a day in the hot tier with the raw stream in a 400-day lake come to about 164 units, and raw retention rises from 90 to 400 days. Whatever your own numbers, the pattern is general: pay for fast access only to what is queried daily.
That is also good news for anyone running analytics. A cheap deep tier is a cheap training set. If you are curious how businesses weigh this sort of trade-off, our earlier write-up on how algorithms sharpen day-to-day business choices covers the decision-making side of the same equation.
What I would watch over the next year
- Whether the schema consolidation holds. The Atlas warns that attributes can be deprecated between releases, so keeping raw originals matters more than any single schema choice.
- Whether pipeline products stay separate from search products. Buyers increasingly ask for routing and redaction before data hits a licence meter.
- Whether privacy tooling matures. A redaction processor for logs in the OpenTelemetry collector is still alpha, and the Atlas notes that keyed hashing is preferable to unkeyed.
- Whether regulators keep specifying retention. US federal memos, Australian-led guidance and India's 2022 directive each set different floors, from 180 days to a year and beyond.
Starting small, the Atlas way
For teams who want to move from reading to doing, the Atlas proposes a gentle ramp. Spend a month listing sources, scoring them and putting health alerts on the ten that matter most. Spend the next writing contracts and test fixtures for those ten, plus a canary event that proves the path works end to end. Use the third to move the three heaviest streams to cheaper tiers, rehearsing each move by replaying a week of raw data to confirm nothing that depends on it breaks. It is a modest plan, and that is its strength.
A note on terminology
It is easy to confuse this field with a neighbour whose name differs by one letter. The Atlas draws the line clearly: SecDataOps builds the data that security work needs, whereas its cousin secures the business data itself. If a vendor slide blurs the two, ask which direction the protection runs.
What to take away
The headline story of the moment is about models. The quieter story is that models inherit whatever the collection layer hands them. Teams that keep a register of sources, write down expectations, keep raw copies in a cheap tier and test their own silence are the ones whose analytics tend to behave. The Atlas lays out a maturity ladder from reactive to optimising, and most organisations, honestly, sit on the first or second rung.
My own view as a watcher of this space is that the next real gains in machine-assisted defence will come from boring improvements in evidence supply, not clever new architectures. If the pipe is trustworthy, even modest analytics look brilliant. If it is not, brilliance is wasted. Another of our recent pieces, on keeping fast visual pipelines from clogging as they grow, makes a surprisingly similar argument about throughput, and it is worth reading alongside this one.