Hey, Luca here! Welcome to a 🔒 weekly essay 🔒 from Refactoring.
Every week I write an article to 170K+ engineers about how make good software together, and interview a world-class tech leader. I also build and maintain Tolaria in the open, publishing my workflows and learnings here.
Recently I have started researching and publishing more case studies about how real companies are doing product development with AI. That’s because I feel that many people are like: “enough with the thought leadership!” — and that includes me!
This time I am doing a deep dive into the AI workflows of the team at Unblocked, which I have known for a long time. I sat down with Dennis and Brandon, who walked me through some of their most interesting processes.
The team at Unblocked is especially relevant because they got skin in the game. They ship a product that turns the team’s code, docs, tickets, and conversations, into actionable context for agents. So they dogfood all of this every day. That’s a strong incentive to work as close as possible to the frontier, rather than doing things “as we have always done” (cit.)
I also love that a lot of what they do is public: they maintain an open-source cookbook with several recipes (skills), which you can fork and copy.
So here is the agenda for today:
🎚️ Dynamic autonomy — scoring runs for risk and confidence, and behaving accordingly.
🧪 Case study #1: vibe check — one skill, two triggers, three exits.
🚨 Case study #2: AI-assisted incidents — humans drive, cleanup can go further
📘 Creating a cookbook around it — interactive vs headless, and how the score moves
📊 Delivery numbers — what the metrics say so far.
Let’s dive in!
Disclaimer: I am a fan of what Unblocked is building and I am grateful to have it as a partner on this piece. You should check it out!
However, I will only write my unbiased opinion about the practices and tools covered, Unblocked included.
🎚️ Dynamic autonomy
One of the most popular ideas about AI workflows is the so-called autonomy slider — that is, for any given process you should design how much leeway to give to the AI, vs how much humans should intervene (and when, and where).
Autonomy is, of course, a spectrum. If perfect autonomy looks like a loop, we talk about humans-in-the-loop to express that some parts are designed for humans to interrupt the AI and do things.
One of the most interesting ideas I got from Dennis and Brandon is about making this autonomy slider dynamic: a lot of their workflows are designed to have runs scored for risk and confidence. Such a score guides how much autonomy the agent gets at that time, and where a human has to step in.
This is a flexible concept that can be applied to many different workflows: shipping code, processing bug reports, intercepting incidents, and more.
In fact, most of these include extremely diverse situations, so it’s tricky to hard-code human gates into the workflows themselves. You constantly risk having either:
Safe work unnecessarily waiting in queue, or
Risky work getting through when it should have been stopped and inspected first.
To make an example, let’s say a bug report spawns from a customer success chat, or from Datadog telemetry. The AI evaluates two questions:
How confident is the AI about this job being real, and being able to do it?
How risky is this change?
This leads to a score that drives the next action:
🔴 High risk, low confidence — just write a ticket and hand it to a human.
🟡 In the middle, roughly 50 to 70% — Slack a person who can answer more questions about the work itself, with the context already attached.
🟢 Low risk, high confidence — write the code directly and open a PR.
The same idea is used on PRs. A change that is evaluated as low-risk can go through without a human review, while a high-risk one puts a human reviewer in the loop.
So, let’s get more into the details of this.
🧪 Case study #1: vibe check
The first process Brandon walked me through was something they call vibe check.