Backups Nobody Has Tested Are Not Backups

A backup job that succeeds every night for a year tells you almost nothing. It tells you the cron fired, the dump completed, and a file landed in object storage. It does not tell you that file will restore, that the restore will finish before your SLA runs out, or that the data inside it is the data your customer actually needs back.

We have operated fleets of per-tenant Postgres databases where the backup pipeline looked perfectly healthy on every dashboard right up until someone needed a restore. The dump was valid SQL. The bucket had the file. Nobody had ever run that file through pg_restore against a clean instance and diffed the row counts. The first real test was the incident.

A backup is a claim. A tested backup is evidence. Everything below is about closing that gap before an outage forces you to close it live, in front of a customer, on a call you didn't schedule.

Why "the job ran" is not evidence of anything

Backup monitoring almost always checks the wrong thing. It checks that the process exited zero, that a file exists, that the file size is roughly what it was yesterday. None of those checks catch the failure modes that actually happen:

• A schema migration ran mid-dump and the backup captured a half-migrated state.
• The dump completed but a foreign-key constraint means it can only be restored in a specific table order nobody documented.
• Encryption keys rotated and the archived backup is unreadable with current credentials.
• The backup captured the database but not the object storage bucket holding uploaded files, so restored rows point at files that no longer exist.
• Retention silently dropped the one snapshot you needed because a lifecycle rule fired a day early.

Every one of these produces a green checkmark on the nightly job and a failed restore three weeks later. The only way to catch them is to actually restore, not to inspect the job's exit code.

The checklist: what "tested" actually means

A backup earns the word "tested" when it has passed all of these, on a schedule, not once:

1. Restores to a clean environment. Not the same box, not a box that still has old data on it. A fresh instance with nothing but the backup file.
2. Restores within your stated RTO. If your recovery time objective is four hours and the restore takes nine, the objective is fiction.
3. Passes a row-count and checksum diff against a known-good reference. Exact counts on core tables, checksums on a sample of rows, not a visual skim.
4. Includes everything the application needs to function, not just the primary database: object storage, search indexes, cache warm state if the app depends on it, encryption keys.
5. Is restorable by someone other than the person who wrote the backup script. If only one engineer can run the restore, you don't have a recovery plan, you have a bus-factor problem.

If a backup has not been through all five, treat it as unverified. Unverified backups are fine as a first line of defense, but they should never be the number you quote to a customer in an SLA.

Run the restore drill on a schedule, not when someone remembers

Restore drills decay the same way fire extinguishers do: nobody checks them until the fire. The fix is a calendar entry, not good intentions. A workable cadence for most SaaS platforms:

• Weekly, automated: spin up a scratch instance, restore the latest backup, run the row-count and checksum diff, tear it down. No human required to trigger it.
• Monthly, semi-manual: one engineer, following a written runbook, restores a backup and brings up a working application against it, including hitting a few real endpoints.
• Quarterly, full drill: simulate the actual incident. Pick a tenant, restore its data to a point in time, and time the whole sequence against your RTO with the person who would actually be on call that night.

The quarterly drill is the one teams skip because it takes real time. It is also the one that catches the failures that matter, because it's the only test that includes the human being confused, tired, and reading a runbook for the first time under pressure, which is the actual condition of a production incident.

Per-tenant backups need per-tenant restores

If you run database per tenant or a hybrid model, a single "we have backups" claim is not granular enough. Backup verification has to answer a tenant-specific question: can we restore this one customer's data, on its own, without touching anyone else's?

We have moved more than a hundred tenant databases between platforms, and the discipline that made those migrations safe is the same discipline that makes backups trustworthy: a reconciliation step that checks the restored data against a known-good source, tenant by tenant, before anyone calls it done. We wrote about the mechanics of that reconciliation step in how to prove your data migration actually worked, and the same report format works for a restore drill. If a restore can't produce a clean per-tenant reconciliation, it isn't ready to be called a backup, it's a snapshot with unverified contents.

A simple reconciliation query for a single-tenant restore drill:

SELECT 'orders' AS table_name, count(*) AS row_count, sum(total_cents) AS checksum_total
FROM orders WHERE tenant_id = $1
UNION ALL
SELECT 'customers', count(*), NULL
FROM customers WHERE tenant_id = $1;
-- run against both source and restored target, diff the results

Run it against the live source and the freshly restored target. Any mismatch is a finding, not a rounding error.

Backups don't cover everything, and pretending they do is worse than admitting the gap

Backups protect against data loss and corruption. They do not protect against a bad deploy that silently writes wrong data for six hours before anyone notices, because by the time you notice, that wrong data is in every backup taken since the deploy. Point-in-time recovery narrows this problem but doesn't eliminate it; you still have to know *when* things went wrong to pick the right point.

This is also where an admin console built for support earns its keep: the faster support can spot and flag anomalous data, the smaller the window between a bad write and the backup you'd actually want to restore to. Backups and observability are not substitutes for each other. A platform with perfect backups and no way to detect a slow data corruption will still lose data, just more slowly and more expensively than one with a bad backup strategy and sharp alerting.

Where this gets more complicated: edge and offline data

Standard backup thinking assumes one system of record with a clean point-in-time snapshot. That assumption breaks in two architectures we build often. Offline-first systems, like point of sale that keeps working with no connection, hold writes on a device until sync, so a backup taken at 2am might be missing transactions sitting unsynced on a register in a store that's closed for the night. Edge deployments on Workers, D1, or R2, discussed in our piece on what actually runs on the edge, scatter state across regions and storage tiers that don't share a single backup mechanism.

In both cases, the honest fix is to define recovery point objective per data path, not per system. The POS terminal's unsynced queue has its own recovery story, usually "the device retains the queue until it successfully syncs," which is a different guarantee than "we back up the central database nightly." Pretending one number covers both paths is the kind of shortcut that looks fine until the specific failure it doesn't cover happens.

The honest tradeoff: testing costs money and time

Weekly automated restores need infrastructure spend for the scratch environment. Quarterly full drills need an engineer's afternoon, sometimes a full day, and they pull that engineer off feature work. For a two-person startup pre-revenue, a full quarterly drill with a dedicated runbook may be more process than the business can afford right now.

The reasonable floor, even for a small team, is the weekly automated restore with a row-count diff. It costs one scratch database and a script, and it catches the majority of silent backup failures: corrupted dumps, missing tables, broken restore scripts. Skip the quarterly full drill only if you've made peace with the fact that your RTO is a guess until the first real incident tells you otherwise.

If backups and durability are part of a bigger platform build, we scope this alongside the rest of the unglamorous half of SaaS engineering, tenancy, billing, entitlements, and migrations, described on what we build, or you can see how the tradeoffs played out on real platforms in our portfolio.