Shipping Fast Changes Your Life ⚡

And what happens to your team when you don't

Last week I discovered that the "10x Engineer" idea had officially become a meme.

I also learned it was originally born out of real findings from a 1968 paper, titled "Exploratory experimental studies comparing online and offline programming performance". The authors showed that the performance ratio between the worst and best programmers was ~10.

I have never been a great fan of this idea. Not only I don't find it particularly useful as a mental model, but it is dangerous too. It easily leads to justifying toxic behaviour, and to creating ridiculous interpretations of the engineering work.

My modest proposal is to shift our focus from 10x engineers to 10x teams.

Great teams make great engineers, while viceversa is not guaranteed.

Everyone who has been in the industry for some time has an intuitive understanding of what separates high performing from low performing teams. If you happened to join a few in both categories, you may have noticed what things look like:

• ⬆️ High performing teams spend their time delivering features and moving the business forward.

• ⬇️ Low performing teams spend their time fixing things, waiting for things (code reviews, deploys, each other) and fighting with things (infrastructure, their tools, each other).

Sounds familiar?

So, how do you create a high performing team? 👇

⚡ Ship fast and often

I am not going to say “that’s it” — but we are pretty close!

After surveying 2000+ companies between 2013 and 2017, Nicole Forsgren, Jez Humble and Gene Kim proved that what separates elite engineering teams from average and poor ones, is mostly how fast and often they ship.

Luca Rossi @lucaronin

Stripe deploys 10 times a day, while maintaining 99.996% availability. This surprises many people, but it's just natural. Continuous delivery brings stability. – Frequent releases = small batches = less risk – Fast releases = fast recovery from failure

Patrick Collison @patrickc

Fun fact: we deployed 3,350 new versions of the Stripe API last year. https://t.co/19Pv2Iu2gk

Now you might argue that correlation is not causation and maybe speed is just the natural byproduct of other things that actually make a team a great team. But I am going to take a stance here and say that no, shipping fast and often is really what you should optimize for, and this alone over time lifts you at a whole another level from those who don't.

Why? 👇

❤️ Shipping is your company heartbeat

In 2013, Intercom wrote that shipping is your company's heartbeat. This is a great metaphor and it stood well the test of time. I believe for two reasons:

1. Just like with heartbeat, if you don't ship you die.

2. Shipping should be just as regular and unremarkable as a heartbeat.

What is less intuitive, though, is how your team's behaviour changes based on how fast or slow you are able to release things in production.

A simple thought experiment 👇

⏱️ Release takes 1 minute

• Feedback loop is instant, no context switch.

• Shipping is continuous — natural extension of development work

• Time to recover is low, few outages

• Confidence is high

⏲️ Release takes 10 minutes

• Feedback loop is good, multiple releases are made every day

• Some context switch happens during/after deploy

• Higher QA investment is justified

⏳ Release takes 1 hour

• Multiple features are batched and released together for convenience

• Failures increase because of batching

• Recovery is slower because of 1) slow deploy and 2) context switch

• More time is spent on QA and tests beforehand to avoid disaster

• "Only in the morning" releases

🗓️ Release takes 1 day

• Deploying becomes a formal process that needs approval

• Only "at start of week"

• Dedicated engineering work is needed to maintain the deployment pipeline

• Frequent outages, pressure on SREs

• Development vs Infrastructure feud arises, teams hate each other, morale plummets

💸 Being slow costs real money

A slow release process costs real money because it makes you need more people.

You will need more QA because diffs are bigger and reviews are longer, you will need an SRE team to manage outages, you will need DevOps to build tools for these manual deploys, and you need will Managers to run this all.

So how do you prevent this? How do you ship fast? 👇

🚚 Automate all the things

Remove as much human intervention from the deployment process as you can, by investing in a solid CI / CD pipeline. That is:

• Continuous Integration — automate the integration of changes into the main software project. This includes running tests and creating artifacts.

• Continuous Deployment — automate the deployment of software changes on a production environment, without the need of any human intervention.

Charity Majors @mipsytipsy

Fast CI/CD is a game changer, a life changer, a completely different profession than slow, arduous, delayed deployments. It is difficult to overstate and difficult to explain to those who have never experienced the difference. And having done so, it is *excruciating* to go back.

Sébastien Cevey @theefer

I can’t overstate how frustrating and counter-productive I find our slow CI/CD (1-2 prod daily prod pushes, hours of deploy pipeline latency), having experienced fast (<10 min) on-demand deploys at the @gdndevelopers. https://t.co/Wv8CY0pS0j

The whole point of doing CI is to being able to do CD. That is the final goal. So:

• ⚡ Keep working on speed — with modern stacks, it is often the case you can literally deploy in 1 minute. But whatever technology you use, do not settle with anything more than 10-15 minutes.

• 🤖 Remove manual checks — or vastly reduce them. When deployment is fast and each release is small, risk and time to recover are so low that most times it is better to release and (possibly) recover than waiting for further checks beforehand.

Also, to help you with this:

• Do not batch releases together. Deploy one changeset at a time to 1) deploy more often, 2) reduce risk and 3) improve ownership.

• Use Feature Flags — decouple deploy from release by using feature flags. They remove bottlenecks and let you deploy without waiting for — e.g. — manual QA.

📚 Resources

• 📑 It is Time to Fulfill the Promise of CI/CD — by Charity Majors. Simply one of the best presentations I have ever seen about CI/CD and development in general. So many great points, very clearly explained. I took many of them 🙏

• 📑 Shipping is Your Company's Heartbeat — by Darragh Curran at Intercom. A memorable article that displays how slowing down your time to production has a devastating effect on your team. Written in 2013, still perfectly on point today.

And that's it for this week! Are you happy with your CI/CD? How long does it take to release to production? Let me know in the comments 👇 or via email!

---

Hey, I am Luca 👋 thank you for reading this far!

Every week I read tens of articles and draw from my own experience to create a 10-minutes advice about some engineering leadership topic.

Subscribe below to receive a new essay every Friday and put your own growth on autopilot!