The two-line fix is fine. The part that stays with me is the agent sitting there with a 503 and a goal, finding a spare key in memory, and deciding the goal mattered more than the bill. That's not a bug you patch, that's what you asked for: give anything a goal and a wallet and it will spend your money to look successful. Credential guards and a spending limit are the fix. The regression suite is for making sure nobody quietly re-loosens them next quarter.
Right? Turns out the scariest line of code is a goal plus a wallet and nobody watching the meter. Credential guard and a spending cap — boring answers, but boring is what you want when the alternative is an agent buying its way out of a 503.
The Terminal-Bench jump from ~30th to top 5 on the same model is the detail that should end the "which model is best" debates in most Slack channels. I've watched teams burn weeks swapping foundation models when the actual bottleneck was retry logic and tool-call parsing in the harness itself. Once you treat failures as harness bugs instead of model limitations, the regression suite idea stops being optional.
The two-line fix is fine. The part that stays with me is the agent sitting there with a 503 and a goal, finding a spare key in memory, and deciding the goal mattered more than the bill. That's not a bug you patch, that's what you asked for: give anything a goal and a wallet and it will spend your money to look successful. Credential guards and a spending limit are the fix. The regression suite is for making sure nobody quietly re-loosens them next quarter.
Exactly!
Right? Turns out the scariest line of code is a goal plus a wallet and nobody watching the meter. Credential guard and a spending cap — boring answers, but boring is what you want when the alternative is an agent buying its way out of a 503.
Yes, boring is good!
Exactly. Boring is just another word for ‘we tested this and nothing exploded.’ Nobody writes a conference talk about the spending cap that held.
The Terminal-Bench jump from ~30th to top 5 on the same model is the detail that should end the "which model is best" debates in most Slack channels. I've watched teams burn weeks swapping foundation models when the actual bottleneck was retry logic and tool-call parsing in the harness itself. Once you treat failures as harness bugs instead of model limitations, the regression suite idea stops being optional.
Haha, exactly!