The Four Measures of Software Delivery Performance 🚚

And why optimizing for speed eventually leads to quality, and stability.

Nicole Forsgren, Jez Humble and Gene Kim surveyed more than 2000 tech companies between 2013 and 2017.

They focused on software delivery, to understand which practices accelerate development, and in turn bring the most value to companies.

Their work had to validate two major theses:

  1. Software delivery performance can be measured in a meaningful way.

  2. Good software delivery performance predicts wider organizational performance — that is, it actually brings value to companies.

They succeeded in both and wrote their findings in a book, called Accelerate.

Accelerate is a cornerstone work for software engineering, as it backs decades of theories with real, indisputable data.

It contains several insights, the most important of which is about speed.

⚡ Focus on Speed

One of the core ideas of the Agile manifesto is that we should break functionality into smaller and smaller items, so we can deliver value faster, and create a better feedback loop.

This principle pre-dates software development, and was actually made popular by the Toyota Production System, or Lean Manufacturing, in the seventies.

Quick delivery proved to be crucial in the automotive industry, and did the same for software — especially in the last decade, with cloud computing and modern toolchains only accelerating the trend.

Most Agile frameworks, however, grew to focus more on the managerial aspects of work (Scrum, Kanban), leaving aside most practical guidance for software delivery.

Accelerate purposely focuses on the latter.

It defines four major metrics for measuring software delivery performance:

  1. ⏱️ Delivery Lead Time

  2. 🚀 Release Frequency

  3. 🔨 Mean Time To Restore

  4. 🔥 Change Failure Rate

These are not the only possible metrics to be considered — but are the ones that have been found to be most correlated with wider, organizational success.

Let's see them in detail, together with target values from high, medium and low performers.

⏱️ Delivery Lead Time (DLT)

Delivery Lead Time, for a given feature or initiative, is the amount of time that goes from the commit to the release in production. It is a subset of the overall Lead Time, which instead covers the whole cycle from ideation to release.

Why measuring one instead of the other?

Lead Time includes analysis and design phases, whose duration can vary a lot based on the breadth of the initiative.

However, once the initiative itself has been broken down in multiple deliverables, the delivery process should move in short and predictable cycles, which are more useful to measure.

Target values:

  • High performers: less than one hour

  • Medium performers: between one week and one month

  • Low performers: between one month and six months

🚀 Release Frequency

Release frequency is how often the organization deploys code for their primary service / application.

It is a good proxy for batch size — that is the average amount of work that gets released in a single time. Reducing batch sizes was one of the key insights provided by the Toyota production system, and we will see that it has a similar role in software development too.

Target values:

  • High performers: multiple times per day (on demand)

  • Medium performers: between once a week and once per month

  • Low performers: between once per month and once every six months

🔨 Mean Time To Restore (MTTR)

MTTR stands for how much time is needed to recover from downtime / issues introduced with the latest release.

Target values:

  • High performers: less than one hour

  • Medium performers: less than one day

  • Low performers: less than one day

🔥 Change Failure Rate

Change Failure Rate stands for the percentage of releases that result in downtime / severe issues.

Target values:

  • High performers: 0-15%

  • Medium performers: 0-15%

  • Low performers: 31-45%

🔍 Analysis Outcome

Analyzing these numbers, and matching them with other organizational indicators (profitability, productivity, market share), researchers got to three key outcomes:

  1. Software delivery performance matters — high performers in software delivery are twice as likely to exceed organizational goals as low performers. They also have 50% higher market cap growth, over three years, compared to low performers.

  2. There is no trade-off between speed and stability — high performers do better at all measures. Actually, delivery speed brings stability, by means of faster recovery from failures.

  3. The gap between high and low performers is incredible — often being 100x on a single measure. You have companies that deploy once every few months, and others of similar size, with comparable tech assets, who do it multiple times a day. That says how early we still are in adopting efficient tech practices.

The overall conclusion is that improving these measures has a crucial impact on both speed and quality of software delivery, and health of the overall business.

Frequent releases enable delivering value faster (speed), which in turn allows for faster feedback from customers, which enables faster improvements on the product (quality).

📚 Resources

Useful readings to dive deeper into these topics:

  • Accelerate: The Science of Lean Software and DevOps — the book that started it all. It's a wonderful reading, recommended to anyone interested in building high performing tech teams.

  • The Machine That Changed the World — the story of Toyota Production System, that kicked-off the lean philosophy and changed several industries forever.

  • Extreme Programming (XP) — one of the most practical Agile frameworks (it actually pre-dates the Agile manifesto), with a strong focus on rapid software delivery. It pioneered several practices like continuous integration, refactoring, and the ten-minutes build.

That's it for this week! What's your experience with software delivery? How do you fare with respect to those benchmarks? Do you plan to improve? Let me know in the comments or via email

Hey, I am Luca 👋 thank you for reading this far!

Every week I read tens of articles and draw from my own experience to create a 10-minutes advice about some engineering leadership topic.

Subscribe below to receive a new essay every Friday and put your own growth on autopilot!