Paul: Today, the spotlight is on Louis-François Bouchard.
He’s the CTO and co-founder of Towards AI and co-author of Building LLMs for Production, with a second book on the way (more on that in an upcoming guest post). He builds AI systems, trains teams on AI adoption and shares practical engineering lessons with 70K+ subscribers on his YouTube channel, newsletter and blog.
Enough chitchat. Let’s get into it 👀 ↓
Can any model write like me? Like, actually write a script I could read out loud without cringing?
That’s what ToneBench, the writing benchmark I released at AI Engineer in New York on October 13, measures. After vibe coding on this project for more than 5 months, spending thousands on API tokens, and making tons of mistakes along the way, I want to share some learnings so you can do better when you build your next evals.
Most scores on ToneBench come from LLM judges. Claude Code and Codex wrote the code. Everything. Whether the scores mean anything came down to a handful of decisions I made that Claude Code never suggested. The results would’ve been TERRIBLE and untrustworthy if I had fully trusted Claude to also work on the judges’ prompts and criteria. So, let’s see what these decisions were and how you can use them yourself.
Quickly on ToneBench: I made it because at Towards AI we write for a living, and I was tired of guessing which model to use.
It’s simple. Every model gets the same 13-20,000-token prompt depending on the task. It has my voice guide, a storytelling guide, the hard rules like target length, and the brief and research for one of ten real scripts from my channel. No tools, no web search. It writes five drafts per script, and four judges plus a few code checks score each one on nine weighted metrics.
With 141 model configurations we measured (many models + different thinking levels), that’s 7,050 drafts and 28,200 judgments. I can’t read them all, so apart from what code scores, every number on the board is the judges’ opinion, so they need to be trustworthy and reliable.
Paul’s rule on this newsletter is to use an LLM judge “only for what code cannot score.” We do the same, but whether a script sounds like me is about as far from code as it gets. So here’s what I’d check before trusting an LLM judge on a subjective task like this.
1. Read every prompt Claude Code writes for your judge
I built ToneBench with Claude Code and Codex, on four 20x subscriptions running for five months, integrating most of the features on a fullvibe coding format and mostly focusing my time on the rubrics, metrics and calibration side of the benchmark; where I believe my “human time” was most worthwhile. Coding agents are fantastic at building what you decide and bad at deciding what to build, which means they can’t make a score trustworthy in this case.
The rubric needs your hands. To be honest, some of what Claude Code drafted for the judges was super weird. Sometimes you don’t even understand what it’s saying, which is bad when that text turns your taste into a number that should mean something. And rules that read well, even ones taken from my own voice guide, can still measure the wrong thing. On our early-October baseline, 119 of 141 models scored above my own scripts on anti-slop, because the rubric punished habits I actually have, like reading an exact figure out loud and repeating it for emphasis. On substance, 102 generations beat me, because any fact outside the research packet counted as invented, even though it was true client insights. Your best outputs should sit at the top of your scale. If they don’t, read the rubric again, and spend time improving it. On October 7, after weeks of iterations on the rubric, I scrapped it all and rewrote most metric sections by hand.
It never suggested the checks I trust most. In four months, Claude never proposed checking whether the same draft gets the same score twice, or comparing the judges with human ratings. I had to think of both and decide to implement both: what if the same draft doesn’t always lead to the same rating? I then, of course, used Claude to implement them and iterate on the “how”, but you, as the owner of the product, need to know the features you need.
It over-invests where it doesn’t matter. I asked for a simple page where people compare two text snippets, and the agents proposed Tailscale, a new paid hosting plan and a security debate about pushing the snippets. At one point I typed “ffs these are just text snippets to compare in my own f***ing team, it’s fine to push it, go.” Then I lost three weeks on that page before realizing it couldn’t tell me what my judges got wrong. More on that later.
Let the agents write the code. Read every judge prompt and make every implementation and measurement decision yourself.
2. Measure the noise before you rank anything
Your typical eval has two sources of noise. The model you are testing, and the judge testing it. It’s easy to forget the second needs measurement too.
For the model, if you only do the task once, it may fool you. I had this intuition, so I regenerated the same task 10 times. From this, I saw wide variance in results across most models. The “worse” the model, the wider the variance in generation quality.
On our current board, we always generate 5 drafts for every single model config to ensure we measure that variance. For example, Grok 4.7 ranks 5th on its best drafts, 29th on average and 67th on its worst. Rating just one draft would mean it lands anywhere between fifth and 67th. But if you average the results across enough generations, you end up with a 29th ranking, which better represents the model’s average quality.
This variance also indicates the model’s quality. Opus 5.5 on xhigh, the best model of the benchmark right now, moves about ±0.9 points between drafts. It always lands first on any given generation, with very little variance.
So, currently, every model writes five drafts per script. This is enough to calculate the per-model variance on our end. But we started with fifty, looked at the spread, and then dropped until we had statistical evidence, and five was enough to see it at an API price we can afford. This is something you’ll need to confirm for your own eval set and tasks, but you need to account for it.
The same uncertainty and variance apply to your judges, since they are also non-deterministic.
I recently wanted to add Gemini as a fifth judge. But before doing so, I needed to know whether it would help. For this, I have a “calibration” set: human ratings my team and I made on how we’d rate generations. We use it to measure accordance, or “calibration,” with our LLM judges, and use it to measure the quality of the rubric we feed them.
So, every time I want to add a new judge or replace one, like Gemini 4, I’ll give it the same prompt as our current judges and measure its rating calibration against our set and against the other judges.
In this case, Gemini didn’t align with our calibration and was rating models almost randomly compared to our current 4 judges (GLM 5.3 Flash, GPT 6.1 Sol, Opus 5.5, Jev). If I had added Gemini, it would’ve shuffled the leaderboard every time we recomputed the scores, making the benchmark totally unreliable. Without the human-made calibration, I would’ve never known this and might’ve added it as a new judge, reducing the benchmark’s reliability.
Those are two examples of noise you need to account for, but there are way more case-specific ones for every single task and system in place. Every component that might add a decision layer, especially probabilistic ones, needs to be measured and compared before being changed.
3. Pass/fail is simpler. If you need a scale, pay for it in examples
Paul recommends pass/fail, because LLMs are bad at 1-to-5 scales. I agree, and it’s usually what we recommend to our clients, too. In June I even told someone on X that a single 1-5 from a judge “mostly measures the judge’s mood.” Well, guess what I ended up building.
Putting “does this sound like Louis?” into binary statements is really hard. Like, really, really hard. We tried, and the results weren’t ideal. They were totally random… We also tried plain categories and straight percentages. What held up best when we judged the same drafts again and again was two steps: the judge quotes evidence, picks one of six levels, each with a description and examples, and only then picks a number inside that level.
That flexibility has a price (in tokens):
Write a lot, and write it yourself. LLM judges need extremely detailed descriptions to understand how to rate it the way you’d like. Claude cannot do this for you, especially on a highly subjective task. Our judges read about 16,000 words per judgment: rubric, brief, research, my script and the draft included. Tone and voice alone (one of the 9 metrics) gets about 1,400 words. I’m still constantly optimizing this rubric with more examples and comparing whether each change improves the judge’s consistency and reliability.
Give scored examples, good and bad, with the why. Each metric has two of my own passages at the top of the scale, then weaker ones, each with a line on what earns that score. A generic passage next to what makes it generic teaches more than any quantifiable rating can on a subjective task. But it also means you need tons of examples for every single possibility to teach the judge what “good”, “bad” and everything in between looks like.
Check your score histogram for magnets. When our rubric used 24 model-written examples, each with a number, Opus gave 52 about eight times as often as 51 or 53, and DeepSeek about forty times as often. We thus added more examples and improved the rubric so it didn’t happen, but you don’t know these things unless you measure them.
Test that a new version separates good from bad. On October 5, a longer, more detailed rubric draft scored the weakest of those 60 scripts 12 to 16 points higher than the live rubric with Opus 5. A shorter version widened the best-to-bottom spread from 31 to 39 points. More detail only helps when it’s the right detail. If you just want to make the rubric as long as possible and as detailed as possible, you risk contradicting yourself and mix up the judges. So measure that too and carefully work on the rubrics iteratively.
Quote before scoring, and hide what the judge shouldn’t use. One or two short verbatim quotes per metric, before the number, are our guard against one strong impression lifting every metric. Judges never see the model name or the word count; when they saw the count, it leaked into other scores. Length, the slop-word list and dashes are scored in code. Coming back to hard-coding evals, we always want to resort to purely deterministic measurements whenever we can.
4. One judge is a vote for its own model
ToneBench started in June with one judge, GPT-5.5, and switched to Claude Opus 4.8 five days later. That’s when I hit a problem one judge can’t solve: Claude models led the board, and I couldn’t tell whether Claude led because it wrote best or because Claude was grading. In July we went to three judges from three labs, and since October 7 it’s four: Claude Opus 5.5, GPT-6.1 Sol, GLM 5.3 Flash (open weights), and Jev, a typed judge from TypeSafe that answers fixed multiple-choice questions and has no model on the board.
Judges never see model names, so whatever they prefer comes from the writing itself. Of course, there are some preferences. Every LLM judge on our panel leans toward its own model. GPT-6.1 Sol gives its own model 7.7 points more than the other judges would predict, and OpenAI 4.1 more. Claude Opus 5.5 gives Anthropic 2.2. GLM’s lean toward Z.ai as a lab is small enough to be noise, though it still gives its own model 1.8.
Our answer is a panel from different families with equal weights, because the preferences pull against each other: GPT-6.1 Sol docks Anthropic about as much as Opus adds. That dilutes the bias without removing it. For labs with five or more configurations, no single judge moves the lab’s average by more than 1.4 points, but on a crowded board that still matters: GPT-6.1 Sol’s lean alone moves OpenAI’s configurations about 8 places on average. So we drop one judge at a time and recompute whenever the scores change. On the October 26 board, removing any one of the four still leaves Opus 5.5 on max and extra high first and second. Even if we remove the Opus judge and only use the three others.
The panel also shows how differently these judges behave (Figure 4):
They disagree on generosity and agree on order. Opus averages 68.0, and GLM 82.4 on the same drafts, yet each judge’s ranking matches the final one at a Spearman of 0.92 to 0.99. It’s like a heart monitor that always reads three beats low. As long as the ratings stay consistent over time and the model ranking order stays the same across our judges, the benchmark works. The actual rating number isn’t so important to match between judges.
The open judge has the smallest lab lean and is the hardest to run. GLM 5.3 Flash judged the whole board for about $65. It’s also the slowest by far, needed 84 schema retries where GPT needed 5, and seems to struggle more to separate good drafts near the top. Great as one voice out of four, and really token efficient, but I would favor “better and bigger models” for judges if the budget allows. Right now, this is a side project, and I can’t spend more than that on judging models, so GLM 5.3 Flash it is!
5. Calibrate against humans, on the same categories as your judges
LLM judges are biased one way or the other, and so are humans, but writing is a human task, so I’d rather our judges be biased toward people. Only people can check that.
I lost three weeks here. My first idea was an arena where the community rates drafts head to head. My team wrote close to 1,000 reviews on 360 pairs, and rating by hand showed the first version was broken: about 90% of the first 50 pairs compared sections with nothing in common. We fixed it, but a deeper problem stayed. Our arena only recorded which passage someone preferred, with no reason and no metric, so it couldn’t tell me which part of my judges was off.
So we’re building something much simpler that I should have started with: a golden set. 240 standalone excerpts of at most 250 words, from strong, middle and weak drafts, plus a few of my own passages and 10 deliberately broken controls. Each gets rated once, by me or one teammate I trust, on 1-to-5 scales for the same kinds of categories the judges score: voice, clarity, slop and overall. Every LLM judge and rubric change will then be compared with it, category by category, in rank order. We already had a “golden dataset” with my real scripts for those 10 tasks, but we didn’t have proper ratings and justifications other than for the rubric itself with small excerpts.
We’ve only just started rating, so no agreement numbers yet to measure the calibration. If I restarted ToneBench tomorrow, I’d begin here, before writing the perfect rubric. This is something we always recommend doing, and, to be fully transparent, I didn’t do it at first because I had no clue if the benchmark would see any traction at all. But it did. If evals and these problems sound interesting, I go much deeper into evals in my upcoming AI Engineering for Production book, which will be out by the end of the year! We’ll kick it off with a live Q&A and some cool perks for all those joining the waitlist. Hope to see you there!
6. Keep it trustworthy over time
An eval you rerun for months will fail if you don’t take the time to survey it properly. Most of our mistakes threw no error and left a slightly wrong board. Only my intuition and querying Claude on what it built and why made us solve them. A few best practices I got out of this project:
Pin the exact judge model, and check who answers. Twice, the model behind our Claude judge’s name changed without an error. In September, a generic “opus” alias reached Opus 5.5 while we were judging with Opus 5, and Opus 5.5 ended up grading its own drafts. Now every judge is pinned to an exact model ID, and a test call must return that model before a run starts. Measure everything, and ask Claude to double-check the monitoring to make sure everything looks fine!
Treat a judge change as a new benchmark. Moving from Opus 4.8 to Opus 5 meant re-judging 4,199 drafts, and 76 of 140 models moved, mostly by a place or two. One changed rubric line costs about $2,350 to re-judge, $3,750 if the models rewrite too. Iterate on small samples: the first run is the cheapest one, but you will discover some issues only when scaling. Be sure to be transparent about how the board may change if you are still working on the rubrics and judges.
Don’t use your judges as validation. All four scored 65 Grok drafts that opened with tool-progress notes instead of a script. The output format was wrong just because of how we prompted it. Instead of flagging this issue, the judges just scored them lower; only our audits caught them. Make sure to double-check the formats and actual outputs and inputs processed. Surprising biases could be breaking your evals. You need to know what’s happening behind the hood.
Run pairwise comparisons in both orders. In our head-to-head check to double-check script-generation quality, the script read second won two times out of three, so a win now counts only when both orders agree.
What I’d do this week
ToneBench measures my voice on my scripts. Use it to shortlist models, then evaluate them on your own outputs. It isn’t a ground truth on which models you should use. Just a good signal on what to test for your own use case when it comes to writing/editing with AI models. The board, more details on the methodology, and the judge data are live and free at benchmark.towardsai.com.
And if you run an LLM judge, do one thing this week: have two people rate a few dozen of your outputs blind, check whether your judge ranks them the same way, and keep those ratings to test every rubric change. It’s the simplest check in this post, and the one I waited too long to run.
If this was useful, I’m finishing AI Engineering for Production, a book that goes much further into evals and what we learned building AI systems with our clients and building this benchmark. We’ll release the book by the end of the year, and the waitlist is open. Anyone joining it will get extra perks when we launch the book!
I hope it helps!
Louis-François
Whenever you’re ready, here is how I can help you
Go from agent user to agent builder. Master the foundations of AI agents and turn fragile demo code into reliable, production-ready systems with my course, Agent Engineering: Building Multi-Agent Systems (made with Towards AI).
35 lessons. Pure foundations from scratch. 4 mini-projects. 2 production systems. A certificate and direct access to me & industry experts in our Discord.
Built for software and data professionals transitioning into AI engineering. Rated 5/5 with 300+ students. The first 7 lessons are free:
Not ready to commit? Start with our free Agent AI Engineering Guide, a 6-day email course on the mistakes that silently break AI agents in production.
Images & videos
If not otherwise stated, all images and videos are created by the author.
