The dashboard shows completed backups every night. Everything is green. Then a database fails, somebody deletes an object or an update makes the application unusable.
The important question is no longer whether a backup exists. It becomes: can we recover a useful service, how long will it take and how much data will we lose?
An agency maintaining solutions for several clients can own dozens of backups without having a repeatable recovery process. The gap usually remains invisible until the worst possible moment.
A copy does not necessarily contain the service
An application is rarely just a database. It may also depend on:
- stored objects or documents;
- environment variables and secrets;
- specific container versions;
- network and DNS configuration;
- queues containing pending work;
- search indexes or vector databases;
- permissions and identities;
- infrastructure defined outside the repository;
- external services and their credentials.
Restoring a table does not prove that document processing works, that an agent retrieves the correct context or that pending webhooks will not be duplicated.
The recovery object must be the whole service, not the backup file.
RPO and RTO without the jargon
Two questions turn a vague expectation into an operational decision.
RPO asks how much work the business can lose. If the latest recovery point is four hours old, everything after it may disappear.
RTO asks how long the service may remain unavailable. It includes detecting the failure, deciding to recover, obtaining access, restoring, validating and reopening traffic.
“We run a daily backup” answers neither question. It may be acceptable for an internal experiment and completely unsuitable for a process receiving orders or client documents.
Five failures a green dashboard cannot show
1. The copy is inside the same failure boundary
A compromised account, a bad policy or an automation error may affect both the service and its backups. Logical separation, restricted permissions and retention policies reduce that blast radius.
2. Nobody knows the sequence
Database first, application next, DNS last. Or perhaps not. When the sequence only exists in one person’s memory, recovery starts with an investigation.
3. Secrets or configuration are missing
The code is in Git and the data is copied, but nobody can recreate identities, certificates or variables. Reproducible infrastructure needs a safe strategy for recovering sensitive configuration.
4. Restoring creates side effects
Replayed events may send an invoice twice, process documents again or trigger old notifications. Workflows need idempotency and a way to distinguish recovered work from new work.
5. Nobody has ever opened the backup
A successful task can produce an incomplete, corrupt or incompatible copy. The convincing test is to restore it and validate the service.
A small but real recovery exercise
You do not need to simulate a regional failure on day one. Start with one client and a limited scenario:
- choose a concrete loss, such as a deleted database;
- declare the expected RPO and RTO;
- assign somebody who did not design the system;
- recover into an isolated environment;
- validate data, access and critical functions;
- record elapsed time and every improvised step;
- turn those steps into automation or documentation;
- repeat until the outcome becomes predictable.
Validation should go beyond “the page loads”. Test a complete transaction: upload a document, run the processing, query the result and confirm that metrics and alerts work again.
Recovery in multi-client operations
When every client uses a different architecture, each recovery becomes a project. A multi-client delivery foundation makes it possible to reuse:
- backup and retention policies;
- restoration procedures;
- infrastructure templates;
- validation criteria;
- incident owners and channels;
- the record of the latest exercise.
Standardisation does not force every client into the same RPO. It applies different service levels through a known process.
Recovery has to be priced
Long retention, immutable copies and fast recovery cost money. So do regular exercises. When these capabilities are promised but not budgeted, they depend on the team’s spare time.
Every maintenance proposal should clarify:
- which components are copied;
- how frequently;
- for how long;
- where copies are held;
- who can restore them;
- which RPO and RTO were agreed;
- when recovery was last tested;
- what remains out of scope.
Not every solution needs instant recovery. Every solution needs an honest expectation.
The question that exposes the risk
Choose one client and ask somebody other than the original builder:
If production disappears today, what do you restore first and how do you prove the service is reliable again?
If the answer starts by searching for access, asking what was copied or reading old messages, the problem is not the backup. It is the absence of a recovery system.
