A 35B model beat a 120B on the same tasks because the harness was tuned for it generic leaderboards would have flipped them. And the key distinction: benchmarks grade the artifact, regressions grade the trajectory.
Nineteen tasks where the 35B beats the 120B. Same harness, same bench. That's the anti-leaderboard point. Hidden verifier after the agent hands back a branch is the right shape. When you pin a new regression from a trace, is it usually the tool-call check or the final-state one?
It depends a lot on what you want to check. But first, you want to squeeze as much as you can with simple code checks and use the LLM judges only when really required.
Fascinating empirical result on model evaluation. It confirms that architectural fitness and narrow latency often trump brute force parameter count in real agentic execution. I explored augmented intelligence and agent design principles here: https://gzambrano.substack.com/p/augmented-mind-llm-agents-second-brain
A 35B model beat a 120B on the same tasks because the harness was tuned for it generic leaderboards would have flipped them. And the key distinction: benchmarks grade the artifact, regressions grade the trajectory.
yes, exactly!
Nineteen tasks where the 35B beats the 120B. Same harness, same bench. That's the anti-leaderboard point. Hidden verifier after the agent hands back a branch is the right shape. When you pin a new regression from a trace, is it usually the tool-call check or the final-state one?
It depends a lot on what you want to check. But first, you want to squeeze as much as you can with simple code checks and use the LLM judges only when really required.
Fascinating empirical result on model evaluation. It confirms that architectural fitness and narrow latency often trump brute force parameter count in real agentic execution. I explored augmented intelligence and agent design principles here: https://gzambrano.substack.com/p/augmented-mind-llm-agents-second-brain
Thanks!
Proof that the harness makes the agent, not just the underlying LLM! Outstanding analysis
Exactly! Thanks ⚡