Quick heads up! On July 22nd I am hosting a webinar on automated incident mgmt with AI, together with the folks at Augment Code.
That’s because I write a lot about my AI workflows with Tolaria, but many people (rightfully) tell me that things are more complex on a real team.
So I invited the team at Augment to share how they transformed their own engineering org, redesigning their main dev workflows. It’s free for everyone and you can RSVP below!
Hi there! A lot of people these days ask me what AI models I use for coding, which I usually explain in my monthly recaps.
Some people also ask me “why” I use one model vs another, and that’s a way more complicated question to answer. AI models are hard to evaluate, because there are several factors to consider, and because, despite what public benchmarks say, each team situation is unique.
So today let’s try to think from first principles about how you should choose what LLM to use for coding, factoring everything in. Also let’s try to find some durable angles, so that you can create a mental model for yourself that stays valid for more than a couple of weeks.
Here is the agenda for today:
🔺 Accuracy / Speed / Cost — the three dimensions of model selection. But they are messier than you might think.
📊 Public benchmarks are not enough — you can’t trust leaderboards, and how your codebase discipline sets your floor.
🧪 How to evaluate AI models — a maturity path from “just don’t” to building your own eval.
🔀 Model routing — not all tasks are equally hard, so stop paying frontier prices for all of them.
🏠 Self-hosting: almost never — cloud-hosted cheap models beat your own hardware, every time.
Let’s dive in
🔺 Model Selection 101
With some degree of simplification, today you have three macro choices:
Top frontier models — e.g. Fable 5 or GPT 5.6
Large open weights models, hosted in the cloud — e.g. GLM 5.2 on OpenRouter
Smaller open weights models, self-hosted — e.g. see DeepSeek 4 Flash via DwarfStar by Antirez
These choices are actually more nuanced than this, because e.g. within frontier models, you may tweak the level of reasoning, or intentionally pick a smaller model that is cheaper and faster, because you feel it is a better fit for your task.
In fact, when judging models, there are three dimensions to consider:
🔬 Accuracy — % of times the AI gets the thing right
⚡ Speed — tokens per second
💸 Cost — cost per 1M tokens (multiplied by tokens required)
To make things even messier, these dimensions are not orthogonal. For example:
A model with higher accuracy may translate into fewer round trips, which may translate into fewer tokens being used.
With many models you can reduce the amount of reasoning to improve speed while reducing accuracy.
So, in most cases, performance is tricky to evaluate. Top frontier models guarantee max accuracy, but do you really need it?
Most people have anecdotal experience of small tasks for which fast + cheap models delivered the same level of accuracy. But how do you recognize them?
Also, it’s trickier than that, because even for bigger tasks you might find that a faster model, while less accurate, iterates faster over mistakes, leading to a lower cost overall. That’s what Jamie Turner, founder at Convex, was already saying last year:
We optimize for fast feedback loops over perfect output. It’s better to get 80% correct code in 2 seconds than 95% correct code in 2 minutes.
Jamie made the case for optimizing for a quick feedback loop, so that developers don’t lose their train of thoughts while waiting for slow responses, even at the expense of a few more errors.
This is even more compelling when you factor in that with the current crop of LLMs, accuracy and cost are not in a linear relationship: a model that is 80% as accurate as the top frontier one might cost 1/10th of it.
Two good examples are GLM 5.2 and Grok 4.5. They both score pretty close to the top OpenAI and Anthropic models, while costing a small fraction of them.
Looking at these numbers is a good way to start some kind of quantitative reasoning around models, but I don’t think it’s enough 👇
📊 Public benchmarks are not enough
I have grown skeptical of public benchmarks, for two reasons: