Before any model scores a customer, flags an anomaly or drafts a recommendation, a dataset has to arrive in good shape. Most disappointing AI projects trace back to that arrival step rather than to the algorithm. DataOps is the discipline concerned with it: shipping and running data pipelines with the rigour of software delivery. The audit below is meant for teams that already have models or analytics in production and want to check, with evidence, whether the feeding pipelines deserve the trust placed in them.
A note on terminology first. The Atlas entry that this checklist leans on states plainly that DataOps is still an emerging term without a formal standard, and that it offers one reading, pipeline operations, among two competing ones. Treat the list as a working instrument, not a certification.
Section A: Is change controlled?
- Pipeline code is in version control. Evidence: every transformation, schedule and alert rule can be traced to a commit. If a notebook on someone’s laptop still feeds a dashboard, mark it failed.
- Changes are tested before merge. Evidence: a pull request runs data tests against a safe copy, not against production. The Atlas warns that a pull-request job with production credentials can run unreviewed contributor SQL, so check what identity your CI uses.
- History can be recovered. Evidence: a snapshot or branch you could roll back to. Rewriting tables in place without snapshots is listed among the Atlas anti-patterns, and it turns every mistake into a permanent one.
- Environments are separated. Evidence: development and staging data are masked or synthetic. Unmasked customer rows in a sandbox are a finding on their own.
Section B: Is every run checked?
The first section covers what your team changes on purpose. This one covers what changes without anyone deciding it, which is the harder half of the job.
Code changes only when someone ships it, yet data changes on every run. This is the run loop, and it is where AI systems quietly lose accuracy. Ask for proof of each item below.
- Freshness, volume and schema are monitored on every table that feeds a model. Evidence: the alert history for the last quarter, not the configuration page.
- Checks are blocking for the critical tier. A warning nobody reads amounts to publishing the bad data anyway. Count your warn-only tests; if there are thousands, the few that matter are buried.
- Loads are repeatable. Replacing a partition gives the same result on the second run, while appending doubles rows. Ask what happens when a job is retried at 3 a.m.
- Late data has a plan. A lookback window or a reconciliation against source counts, written down.
Section C: Are the statistics honest?
This is where an AIOps-minded team can be fooled by its own tools. Three audit questions come from the Atlas material and are worth asking of any anomaly detector.
- Does one baseline cover a process with weekly seasonality? A limit computed across all days can be so wide that a partial weekday load passes. The Atlas lab reports a load roughly 19% below a normal weekday that the all-days limits missed and a weekday-only baseline caught.
- What is the false-alarm budget? A three-sigma limit has about a 0.27% chance of firing on any point, so hundreds of checks per day produce noise by arithmetic alone. Teams that never measured alert precision tend to mute the monitors.
- Could a learned baseline be absorbing a slow decline? Inferred thresholds follow the data downwards. Pair them with a fixed floor on the tier-one tables.
A related caution: tests generated automatically by a tool describe the data as it currently is, not as it should be. They need a human review before anyone calls them a specification.
Section D: Who is accountable?
- Each critical dataset has a named producer and a contract covering fields, meaning, quality rules and delivery expectations. Structural checks alone, such as schema compatibility, pass even when the meaning of a status value changes.
- Incidents are logged with detection time. The Atlas worked example splits downtime into detection and repair, and finds that three quarters of the total was the wait before anyone noticed. Calculate your own consumer-detected share: the proportion of incidents first reported by users rather than by monitors.
- Tiers are explicit. Critical, important and exploratory pipelines get different rigour, so the effort lands where an error is expensive.
Scoring the audit
Scores only mean something if they come with artefacts. A screenshot of a dashboard is not evidence; an export of last quarter’s alerts, a pull-request log and a list of owners are. Ask the engineers who run the pipelines to present the proof, and ask someone from the consuming side, perhaps an analyst or product owner, to challenge it.
Tally one point for each item with evidence you could show a sceptical colleague. Fewer than eight points out of fourteen, spend the next month on Sections A and B before touching model quality. Between eight and eleven, concentrate on Section C. Above that, your remaining work is the slow kind: reducing consumer-detected incidents quarter by quarter.
If you are reviewing how AI features reach decisions in your own organisation, our earlier article about AI and day-to-day business choices makes a handy companion read, and for a softer introduction to what these systems do with inputs, see the plain guide to how image classification reads a picture. For the definitions, the scope boundaries and the maturity ladder behind this audit, consult the DataOps page in the Ops Atlas from xOps School, whose own ladder is explicitly described as the page’s framing and not an industry standard.
Run the audit once, file the evidence somewhere reviewers can find it, and repeat it in a quarter. Expect the second pass to be faster, because the owners and alert exports will already exist. The trend matters more than the first score.