Too much of this article is based on a great misunderstanding of what staging environments are for. Hint: they're need about system complexity.
No, not everybody needs a staging environment. But if you are getting no real value from your staging environment, then take the time to understand its purpose BEFORE scrapping it.
When understanding staging environment's purpose the critical questions are:
- What's the potential impact of a bug?
- Who's doing the testing?
- How are the tests performed?
- What's being tested?
First off let's deal with the false connection between small atomic changes and low impact changes. Risk is measured in two dimensions: probability and impact. Small changes have a low probability of causing a bug but the impact might still be huge. Consider changing just one character from a jet engine's firmware. If that one character change causes a bug, people might die. So the context of the software is what's important, NOT the number of lines of code changed. When deciding if you need a staging environment, the impact of a potential bugs is most important.
Next, staging environments are often used to allow non-developers to take part in the testing before production. Larger projects and organisations may have a dedicated test team and software testers are often not developers. Squeaky's alternative to this, using production as if it's stage by hiding "risky" features behind a feature flag, is viable in only where bugs are all low impact. Even then different users have different tolerance to bugs. So staging environments are commonly used to limit who can see potentially embarrassing missteps in design.
Then you can look at the type of testing you want to perform. In may domains minor bugs can cause very costly drift over time. Imagine a banking app that accidentally miscalculated 10% of transactions by $0.01. Just running through a checklist of tests for 30 minutes not trap that. It may be necessary to run the software a week or longer before you're confident that the changes were "safe". Time is an important factor of staging environments.
Likewise the suggestion that merging code "should automatically" deploy to staging is questionable. It really depends on the testing being done there. In many contexts, the testers need to ensure they are testing a consistent platform. In which case stage environments are change controlled just like production. Indeed its hard to justify the use of a staging environment when you don't know which version of the code it was running when you did your testing.
Finally, the claim that you can dramatically increase the feedback loop by removing the staging environment is only true if you only ever do developer testing 🤠. The real limit on that feedback loop is not the staging environment at all, it's the handover from developer to tester [and back if required]. If nuking your staging environment enables code to get from developer to production inside 10 minutes then you were never really using stage for its intended purpose.
I don't deny that many companies do not get really good value from their staging environment, but in a sizeable proportion of those cases they would be better to improve the testing they do in staging, rather than scrap it all together.
So the stakeholders test their applications directly in production? what if they want to make "silly" or "corner case" tests that might negatively affect production?
One thing that adds to complexity of staging: data required for proper testing/bug fixing needs to be fairly similar to prod in certain cases, eg when a bug is limited to certain user, role, etc. This implicates a need to anonymize data that is pulled from production in order not to raise security issues, and that substantially adds to the cost. Highly relevant when you process personal data and/or have certain regulations to comply to.
I like how you clearly wrote this post because of the Squeaky article, and paraphrased or copied parts of it, yet only dropped a link at the end and criticised it for being naive...at least be honest when you're trying to piggy back off the success of someone else's content.
Hey I'd love to know what tool do you use for drawing the diagrams?
Too much of this article is based on a great misunderstanding of what staging environments are for. Hint: they're need about system complexity.
No, not everybody needs a staging environment. But if you are getting no real value from your staging environment, then take the time to understand its purpose BEFORE scrapping it.
When understanding staging environment's purpose the critical questions are:
- What's the potential impact of a bug?
- Who's doing the testing?
- How are the tests performed?
- What's being tested?
First off let's deal with the false connection between small atomic changes and low impact changes. Risk is measured in two dimensions: probability and impact. Small changes have a low probability of causing a bug but the impact might still be huge. Consider changing just one character from a jet engine's firmware. If that one character change causes a bug, people might die. So the context of the software is what's important, NOT the number of lines of code changed. When deciding if you need a staging environment, the impact of a potential bugs is most important.
Next, staging environments are often used to allow non-developers to take part in the testing before production. Larger projects and organisations may have a dedicated test team and software testers are often not developers. Squeaky's alternative to this, using production as if it's stage by hiding "risky" features behind a feature flag, is viable in only where bugs are all low impact. Even then different users have different tolerance to bugs. So staging environments are commonly used to limit who can see potentially embarrassing missteps in design.
Then you can look at the type of testing you want to perform. In may domains minor bugs can cause very costly drift over time. Imagine a banking app that accidentally miscalculated 10% of transactions by $0.01. Just running through a checklist of tests for 30 minutes not trap that. It may be necessary to run the software a week or longer before you're confident that the changes were "safe". Time is an important factor of staging environments.
Likewise the suggestion that merging code "should automatically" deploy to staging is questionable. It really depends on the testing being done there. In many contexts, the testers need to ensure they are testing a consistent platform. In which case stage environments are change controlled just like production. Indeed its hard to justify the use of a staging environment when you don't know which version of the code it was running when you did your testing.
Finally, the claim that you can dramatically increase the feedback loop by removing the staging environment is only true if you only ever do developer testing 🤠. The real limit on that feedback loop is not the staging environment at all, it's the handover from developer to tester [and back if required]. If nuking your staging environment enables code to get from developer to production inside 10 minutes then you were never really using stage for its intended purpose.
I don't deny that many companies do not get really good value from their staging environment, but in a sizeable proportion of those cases they would be better to improve the testing they do in staging, rather than scrap it all together.
So the stakeholders test their applications directly in production? what if they want to make "silly" or "corner case" tests that might negatively affect production?
One thing that adds to complexity of staging: data required for proper testing/bug fixing needs to be fairly similar to prod in certain cases, eg when a bug is limited to certain user, role, etc. This implicates a need to anonymize data that is pulled from production in order not to raise security issues, and that substantially adds to the cost. Highly relevant when you process personal data and/or have certain regulations to comply to.
I like how you clearly wrote this post because of the Squeaky article, and paraphrased or copied parts of it, yet only dropped a link at the end and criticised it for being naive...at least be honest when you're trying to piggy back off the success of someone else's content.