“It worked in staging” is not an explanation. It is a signal that two supposedly comparable environments are validating different realities.
The same commit can fail in production because of data, permissions, load, configuration, deployment order or an external dependency. The more handcrafted each client’s infrastructure is, the harder it becomes to identify which difference matters.
The false comfort of a green environment
Staging is useful when it reduces uncertainty. It loses value when it only confirms that a release starts with little traffic and clean test data.
A successful pass does not necessarily prove:
- the migration supports real volume;
- identities have equivalent permissions;
- required variables are present;
- integrations accept production traffic;
- rollback remains compatible with the data;
- alerts will reach somebody;
- retries and concurrency are safe.
The question is not whether staging is identical. It rarely can be. The question is whether it reproduces the conditions that could invalidate the change.
The differences that usually hurt
Data
Test data is small, clean and coherent. Real data contains old records, missing fields, huge documents, unexpected encodings and states nobody remembers.
A migration should be tested with representative volume and distribution without indiscriminately copying personal data.
Identity and permissions
Staging may use an overly broad credential. Production uses a restricted identity and the change fails when writing to a bucket, reading a secret or invoking an API.
Policies should come from the same template and vary through explicit parameters.
Traffic and concurrency
A manual test confirms one path. A hundred concurrent requests expose races, connection limits, timeouts and non-idempotent operations.
You do not always need to reproduce the full peak. You should test the load assumption that justified the architecture.
External dependencies
Payment sandboxes, AI models and third-party APIs may have different limits and responses. Production also introduces allowlists, domains, certificates and real quotas.
Every dependency needs a pre-flight check and defined behaviour when it stops responding.
Change sequence
Code and database updates cannot always happen atomically. If a new release requires a schema the old one cannot understand, rollback may no longer be safe.
Backward-compatible changes and phased deployments reduce that risk.
Parity does not mean duplicating cost
A permanent production clone may be unnecessary. Useful parity focuses on:
- the same provisioning method;
- the same major versions;
- the same deployment mechanism;
- the same identity model;
- the same critical policies;
- equivalent observability;
- documented and reviewable differences.
Capacity may differ. The process should not.
Keep a difference register
A small table prevents exceptions becoming invisible:
| Dimension | Staging | Production | Risk | Control |
|---|---|---|---|---|
| Data | synthetic sample | full volume | slow migration | volume rehearsal |
| Access | internal VPN | public traffic | different rules | connectivity test |
| Capacity | minimal | autoscaling | hidden limits | load test |
| Integration | sandbox | real account | quotas and permissions | smoke test |
If a difference has no reason, remove it. If it has a reason, add a control that compensates for the risk.
Production needs its own validation
Staging does not replace safe delivery. Production needs its own defences:
- pre-change checks;
- gradual rollout when risk justifies it;
- automated smoke tests;
- metrics compared with the baseline;
- an explicit rollback criterion;
- one person accountable for the decision.
A small observable change is often safer than extensive pre-validation followed by a blind leap.
A useful agency exercise
Choose the latest failed deployment and record which environment difference caused it. Then ask:
- was the difference necessary?;
- was it documented?;
- could the pipeline have detected it?;
- does it affect other clients?;
- can the solution become a common template?
When every client has a different staging environment and process, another checklist is not enough. The answer is a common way to onboard and operate clients.
