Most data loss is not a disk failure. Storage providers are very good at not losing bytes. Data loss is a cleanup job with a wrong prefix, a cascade delete nobody traced, a migration run against the wrong environment — authorised operations doing exactly what they were told.
Backups help with that only if you notice in time. If a job quietly deletes a class of objects and nobody looks for six weeks, a 30-day backup window is decoration.
Make deletion physically hard
The control that actually works is making recent objects undeletable at the storage layer. An object lock with a retention window means a delete of anything newer than the window fails — not because your code checked a permission, but because the storage refuses.
We run a seven-day lock on the bucket holding customer documents and generated plan imagery. It has a real cost: cleanup code that tries to remove a file it just wrote gets an error, and that code has to be written to expect it. We consider that a feature. The error surfaces at development time, on a path where a developer is already looking.
A guard rail you never bump into is a guard rail you cannot prove exists.
What we check
- Can a delete of a fresh object succeed? It should fail. If it succeeds, the lock is not configured.
- Does a restore actually restore? Untested backups are a belief, not a control.
- Does the checksum of a restored file match what was recorded at backup time?
- Is there a written runbook, and has someone other than its author followed it?
That last one catches the most. Runbooks written by the person who built the system encode assumptions that person cannot see. The first time someone else runs it is the first real test.
Retention is a product decision
How long you keep things is not an infrastructure question, it is a promise to the customer, and it should be written where the customer can read it. If event photos are purged 30 days after an event unless someone pays to keep them, that belongs in the product copy, in a warning email before it happens, and in a scheduled job that does exactly what the copy says.
The failure mode here is drift: copy that says one thing, a cron job that does another, and a support conversation that discovers the gap. Whenever we write a retention rule we write the test that asserts the job matches the promise.