7 Comments
User's avatar
Halfcycle's avatar

Step 7's regression suite, past failures frozen into tests, is the part most homemade harnesses skip and the one that compounds. The question we kept running into while building Halfcycle: what about the failures someone else's project already hit? A suite grown from your own traces only knows your own traces, so every team relearns the same lessons. Did any of Decode's benchmark failures come from other people's runs, or all from yours?

Paul Iusztin's avatar

Decode is just an educational harness, not available to multiple users, so all the examples are just from my interactions with it

Halfcycle's avatar

Makes sense for a teaching harness, and your own traces are the right place to start. The step after is the one we find interesting: once a harness runs for many teams, the cause and the fix from one project can reach the others automatically. Most of those problems then never show up in your project, and the ones that do arrive with the answer attached.

Andrii Tretiak's avatar

"Same model, better harness" matches what I saw in my own setup. The biggest jump for me wasn't a new model, it was cutting my agent system from 42 agents to 13, moving the fixed steps into hooks and scripts, and loading skills only when a task needs them. Less stuff in the context, and the agents stopped mixing up rules from different tasks.

Paul Iusztin's avatar

yes, that's the gist!

Shrey Malhotra's avatar

The harness moving LangChain's agent from outside the top 30 is a great proof point. Do you think product teams will start owning the harness, or does it stay an engineering concern? I'm Shrey, CPO at CambrianEdge.ai.

Paul Iusztin's avatar

I think it depends on the scale / use case of your team. But yes, I think the scenarios where you need to build custom harnesses will increase