How to do QA in 2024 🧪
A massive guide about this confusing but crucial practice.
Every week I get tons of questions from readers about topics I should cover in the newsletter. Some of the most frequent ones are about QA — in fact, I believe I receive at least one per week.
Are these questions about specific niche optimizations or tactics? No, they are about basic things: who should do QA, what good QA looks like, or whether you need QA at all.
You might think these get asked by fresh managers or young founders — but no, they often get asked by folks who have 10+ years of experience, who are not afraid to admit they are just… confused.
There is little consensus around the role of QA in modern engineering teams, which is the reason why I am writing this piece.
Recently, we even held a Mastermind session about it, which brought together experienced engineering leaders and allowed me to collect several experiences, and reflect on them for this article.
So, this is a full guide (3500+ words!) that explores everything I know about QA: principles, practices, tradeoffs, team organization, technology, and tools.
Here is what we are going to cover:
🧠 QA is a mindset — the difference between QA and testing.
🏆 The value of testing — figuring out the ROI of your tests.
💸 E2E testing is expensive — a basic premise that you can’t ignore.
⬅️ Shifting quality left — getting the foundations right before you get to QA.
🛣️ QA vs product journey — how QA changes at different stages, from zero to hero.
💪 QA in your team — who should do QA? And how? Handling tech and duties.
💼 Using external services — can you delegate QA?
Let’s dive in!
🧠 QA is a Mindset
If you can’t find a satisfying definition of QA online (I couldn’t), that’s because QA is a mindset, rather than a specific process.
QA is making sure you meet an expectation of quality for your product, and in particular that you don’t go below that quality. Some people think it’s simply “we don’t ship bugs”, but things are more nuanced:
If you ship a bug on something no one’s using — it obviously doesn’t matter.
If you break your most critical workflow and you have hundreds of paying customers, it matters.
If you break your most critical workflow but you only have a few customers, and they don’t pay or don’t pay a lot, it may not matter, too.
This awareness about where the value is, is the reason why we talk about Quality Assurance, as a broad function, as opposed to simply Testing.
Good QA requires sorting out what’s really valuable from what is not, and while you might argue this applies to testing in general, QA does this from a product/business standpoint, rather than a pure engineering one.
This mix of product and tech duties is also why QA is confusing to handle: should PMs do it? Or should engineers? Or should you have dedicated people for it?
We’ll get there, but for now, let’s talk about testing 👇
⚖️ QA vs Testing
When we talk about QA, we usually refer to a specific approach to testing—testing the product like a final user. This means actually using features and workflows to identify issues.
Now, in software, you have three (main) categories of tests: unit, integration, and end-to-end. With some degree of simplification, QA is about end-to-end testing. This is also called black-box testing, because it can be performed without much knowledge of tech internals — as opposed to white-box testing, like unit and integration, that is more coupled to the specific tech.
Honestly I am not a big fan of this definition, because, to me, this looks more like a continuum than black/white 👇
If you imagine tests as a pyramid with unit tests at the bottom and end-to-end tests at the top, the higher you go, the more complex the workflows you're testing, and the darker this box becomes because you can ignore more layers below.
But how is this useful? Are tests less or more valuable based on this? 👇