Do you really need a Staging environment? 🚢
Or does it bring more problems than it solves?
Most companies I know use staging or pre-production environments to test recent changes before they are released to real users.
This lets them discover more bugs, but it also increases the cost and complexity of their delivery process.
In recent years, anything that delays or makes the release slower, like testing environments and complex branching structures, have been put under discussion, with more and more teams gravitating towards simpler setups.
That’s because the faster and the more often you release software, the less risk you take, and the more effective you are. Top-performing teams know this, and they adopt a continuous delivery approach that favors atomic releases and always-deployable software.
This article isn’t meant to be a hot take, but rather an objective analysis of the upsides and downsides of using a Staging environment. The goal is to help you get clarity about what is best for your product and your workflow.
This is also a multi-player effort that involved several folks from the Refactoring community like Alex, Zach, and Yurii, who shared their takes and helped heavily with this.
We are going to cover:
📖 What is a Staging environment? — definitions first.
🚢 The problems with Staging — technical and process issues.
⛵ How to live without Staging — while minimizing release risk.
🌊 When to keep Staging — what are the scenarios where it pays off.
Let’s dive in 👇
📖 What is a Staging environment?
Staging is a safe space where teams can test their software in an “as-close-to-production-as-possible” environment, before the actual release.
It was originally conceived in the era of software shipped in boxes, when there wasn’t a controllable production environment to test or gather data from.
If you use Staging, your release process might be something like this:
Merge feature branch — on the Staging branch (e.g. “develop”), after running automated tests.
Release on Staging — this should be automated after the merge.
Test on Staging — QA and possibly feedback from product and non-tech stakeholders.
Release to production — merge the Staging branch to production and release.
There are alternatives, of course:
If you do trunk-based development, you may not have a “develop” branch that gets merged to production, but rather push feature branches to staging, and after testing merge the feature branch directly to the trunk.
You may have more than one intermediate environment. Some teams have separate envs for QA, pre-production, and more. For the sake simplicity, we will use the word “Staging” to mean any of these intermediate envs.
So, the role of Staging is clear. It is a shared place where you do some testing before the release.
What is bad about it? 👇
🚢 The Problems with Staging
Staging environments suffer from two main issues:
They are often not reliable — because it is hard and expensive to keep them at parity with production.
They make releases many times slower — by introducing an additional release level and batching changes together.
Let’s see why these are serious problems.
1) It is hard to keep Staging at parity with production
For Staging to be useful, it has to catch a special kind of issues that 1) would happen in production, but 2) wouldn’t happen on a developer's laptop.
What are these? They might be problems with data migrations, database load and queries, and other infra-related problems.
To make Staging catch these issues, you need to keep it at parity with production on data and infrastructure. This is hard and expensive — think about it, if it wasn’t so, you would just spin dev environments that look like prod.
The whole point of having a single, shared environment for testing instead of many individual ones is that the latter would be too expensive to maintain.
The way I see it, fundamentally, this is a resources management problem. If I wouldn't be looking at costs the dev environments could be designed and made powerful enough to satisfy all needs.
— Alex Stoia, CTO at Innertrends
To maintain such parity, you bear two costs:
Infrastructure cost — the cost of running the hardware.
Upkeep cost — the maintenance cost to keep environments aligned.
While there isn’t much you can do about the former, the latter is less of a concern when you use infrastructure-as-code and modern DevOps tech.
If components are built from scratch to be aware of their environment, and the component's infrastructure code and configuration are both hierarchical, it's not much additional effort to maintain a staging environment. I have on more than one occasion "spun up a quick burner environment" for a project only to tear it down and not feel any regret on wasted time.
— Zach Wolfe, Senior Software Engineer at Amazon
In my experience, however, most companies cut corners on this and end up with Staging setups that look nothing like production. For example, they may hold a small fraction of the database, or run on totally different instances.
This defeats the purpose of Staging and makes it unable to do its job.
2) Releases are many times slower
There is no way around it. This might be for a good cause, but it’s important to understand the full consequences.
When your release time is fast — like, 10 minutes fast — good things happen: