Ask an AI team what their model-monitoring stack measures and the answers come quickly: drift, latency, precision, cost per inference. Ask the same team what their test suite measures and the room goes quiet, because the usual answer is a raw count of checks and a green-or-red blob on a dashboard. That gap is what TestOps exists to close. The term describes running an automated test suite as a production service — one whose users are developers and whose output is a fast, dependable pass-or-fail verdict on every change. This walkthrough sets one up in seven steps, each with the evidence you should collect before moving to the next.

Step 1: Measure Before You Touch Anything

Baseline first, for the same reason you would never tune a model without a holdout set. Four numbers matter in the first week, all computable from existing CI logs if results are stored per attempt. False-red share: the proportion of runs that failed on noise rather than defects. Time to verdict at the median and ninetieth percentile. Main-branch green rate: how often the primary branch is actually usable. Cost per verdict: billed CI minutes divided by the number of verdicts produced. Slack’s mobile team published a striking before-and-after on one of these — once flaky results were handled properly, their main-branch green rate moved from 19.8 percent to 96 percent. Numbers like that turn arguments into arithmetic.

Step 2: Build the TestOps Results Store

A verdict that cannot be queried later might as well not have happened. The core asset is a results store keyed by a stable identifier — conventionally the classname and test name — with the commit hash and the attempt number attached to every record. With that schema, flakiness stops being folklore: select every case where one identifier produced both outcomes on one commit and you have your confirmed-flake list, mechanically derived. The same store powers trend lines, quarantine reporting and the analytics in later steps. Treat vendor lock-in as a data problem rather than a tooling problem: as long as attempt-level results sit in an exportable form, the runner can be swapped.

Step 3: Set Stage Budgets and Enforce Them

Speed is a design parameter, not a lucky accident. A workable ladder of budgets looks like this:

• On the laptop: seconds, for the inner loop.
• Pre-merge fast stage: at most ten minutes, which DORA identifies as the sensible ceiling for feedback developers will actually wait for.
• Merge queue: a short verification run so merges stop breaking the branch behind them.
• Post-merge: the full suite, sharded across workers, landing near thirty minutes.
• Nightly: the slow, expensive, low-frequency checks.
• Post-deploy: smoke checks, synthetics and canary analysis watching real traffic.

Whenever a late stage catches a defect, the fix is a faster check upstream that would have caught it sooner; migrated downward this way, the ladder tightens itself. One practical trap: GitHub’s merge queue only triggers on the merge_group event, and pipelines that never subscribed to it mysteriously “skip” the queue.

Step 4: Separate Noise from Signal, Mechanically

With a results store and a budget ladder in place, suspected flaky checks can be pulled from the deciding path, each one carrying a named owner, a tracked issue and an expiry date — never an unbounded backlist. A new check should prove itself in a burn-in period before it gains the power to block anyone, since a large share of flaky checks are unreliable from the day they are first committed. The lifecycle is explicit: burn in, then gate, then suspect, quarantine and finally delete.

Step 5: Put a Number on Trust, the TestOps Way

An SLO makes the service contract real. The Ops Atlas TestOps entry suggests a defensible formulation: at least 98 percent of verdicts over a 28-day window should be true verdicts One defensible formulation: at least 98 percent of verdicts over a 28-day window should be true verdicts — neither a false red nor a hidden defect. On 1,200 runs that budget allows 24 false reds, which sounds generous until the compounding arithmetic of flaky suites is laid bare. Weekly review of the Step 1 metrics against this target is the heartbeat of TestOps; the numbers are few, but they are the ones that describe whether the verdict stream deserves the authority it has been given. Vanity measures — total test count, coverage percentage, post-retry pass rate — describe activity, not trust, and belong in nobody’s review.

Step 6: Let a Model Pick the Tests, Then Check Its Work

Test selection is where an AIOps mindset pays directly. Meta trained a model to choose which tests to run per change; running roughly a third of the tests downstream of a commit, it still caught upwards of 99.9 percent of the regressions — and, importantly, the model was trained on de-noised data, because one that learns from flaky results learns the noise. The honest caveat: beyond selection, claims for self-healing checks and AI-written tests do not yet have independent evidence behind them. Log every automated heal, keep deletion and gate policy human decisions, and treat vendor demos accordingly.

Step 7: Keep the Gate Honest

The last architectural detail is the least glamorous and the most commonly botched. The gating job must run even when earlier jobs fail — in GitHub Actions terms, a guard condition such as if: !cancelled() — otherwise branch protection sees no required status, and red changes merge themselves straight to main. Wire the gate against the results store, not the CI log, and test the failure path deliberately once.

Where This Fits the Wider Picture

Teams that already practise AIOps will recognise the shape: instrument first, define service levels second, automate judgement third, and keep humans on the decisions that matter. The same reasoning that powers how AI improves business decision-making applies here — models earn authority through measured error rates, and so do test suites. If you are assembling the broader skill base, a DevOps engineer roadmap covers the pipeline fundamentals this discipline assumes, and for the classification problems inside test selection, the guide to how AI looks at pictures is a gentle introduction to the underlying ideas. Seven steps, seven sources of evidence, one service worth trusting.