Machine-learning platforms accumulate things that are expensive to recreate: labelled datasets, feature tables, trained weights, evaluation sets, prompt libraries and the pipeline definitions that tie them together. Protecting that estate is the territory of BackupOps, a term the Ops Atlas describes as informal rather than a recognised discipline, and which it reframes as restore engineering. This audit checklist is written for AI and automation teams who want to find out, in an afternoon, whether their copies would actually bring a model platform back.
Work through the sections in order and record a plain yes or no against each line. Treat each “no” as a task awaiting a date.
Section 1: inventory and ownership
- Every asset that matters (training corpora, feature store, registry of model artefacts, vector indexes, experiment metadata) appears on a list.
- Each listed asset has one named owner, a written policy and a place where evidence is kept. The Atlas puts the failure mode bluntly: when everyone owns backups, no one does.
- Assets are ranked in tiers, so that the top few get the strictest treatment first.
Section 2: the four-question test for each copy
The Atlas proposes four questions for judging a backup, and they translate well to ML assets. Put them to every tier-one item.
- Is the copy independent of production, meaning a different failure domain, account and set of keys?
- Is it intact, so that checksums or manifests confirm nothing was damaged in transit or at rest?
- Is it fresh enough that the work lost since it was taken stays tolerable?
- Can it be restored inside the time you can afford to wait?
An asset that passes all four is what the Atlas calls proven recoverable. Anything less is a hope, however many green ticks appear on a dashboard.
Section 3: the arithmetic nobody does
Two sums catch most surprises. First, loss window: if a nightly job starts at 01:00 and its output only reaches the isolated vault at 05:00, a failure just before the transfer finishes loses roughly 28 hours of work, which breaks a 24-hour target even though every run succeeded. Second, recovery time: a dataset of 4 TB read at 250 MB/s by a single stream takes about 4.4 hours, whereas four parallel streams bring it near 67 minutes before any validation step. Run both sums for your largest feature store and your biggest corpus, and write the results beside the targets.
Section 4: separation and credentials
- At least one copy sits where production credentials cannot delete it.
- No automation agent, orchestrator or scheduled job holds delete or unlock rights over that copy. The Atlas is explicit that language models can summarise logs and draft runbooks but cannot prove that something is recoverable, and it adds that no agent should be given such rights.
- Encryption keys survive the same incident as the data, held in escrow outside the production environment.
- Replicas and snapshots are not counted as backups unless they live in a separate account or domain; a replica copies a deletion as faithfully as it copies a file.
Section 5: restore testing ladder
The Atlas describes four rungs, each more convincing than the last: integrity checks, sample restores, full restores into a sandbox, and timed exercises. Mark which rung each tier-one asset has reached. For model platforms, a sample restore might mean pulling one registered model plus its training manifest and loading it in a clean environment. A full sandbox restore would rebuild the feature store from backup and compare row counts and a handful of known values.
Section 6: measures that mean something
Job success rate is described in the Atlas as a vanity metric. Its worked example pairs a near-perfect job success rate with a tiny share of assets that anyone had ever restored successfully. Prefer coverage, loss-window compliance, restore-test currency, isolated-copy coverage and a composite proven-recoverable figure. You can read the fuller list in the Ops Atlas BackupOps entry, where the proposal is laid out with its caveats.
Section 7: sizing the bill
Training data grows, and retention multiplies it. The Atlas gives a rough illustration: 10 TB of protected data, changing 2 percent a day and held for 35 days, ends up as about 17 TB in primary backup storage plus the same amount again in the isolated vault, which is around 3.4 times the data being protected. Run this multiplication for your own corpora before choosing a retention period, and remember that cold storage tiers can carry minimum charge periods of 90 days, so short-lived experiments are not always cheaper to park there.
Connecting the audit to the rest of your AI estate
The same discipline of evidence over assumption shows up in other pieces on this site. A look at the ways AI improves business decision making only works if the underlying data survives, and an article on scalable, high-speed visual systems raises the question of where the image archives and trained weights are kept when something breaks. Neither piece is about recovery, but both rest on it.
Scoring and next steps
Count the yes answers in sections 1, 4 and 5, then total the cost figure from section 7 as a sanity check on whether your plan is affordable at the scale you expect in a year. If fewer than half are yes, schedule one hand-run restore of your single most valuable asset within the next month, and publish the result to the team whatever it shows. The Atlas suggests exactly this as an early step: restore one tier-one item by hand, then report the first proven number. If the restore fails, treat it as an incident for that tier rather than an embarrassment to hide. The only unacceptable outcome is not finding out until the day the copy is needed.