Arena AI Release Notes

We’ve updated how the Bash Recovery score is calculated on the Agent Arena signal leaderboard and are now naming this signal, Bash Reliability. There are two changes: how we detect bash failures, and how we turn them into a score.

• Detecting failures. Previously, we used a simple set of rules to decide whether a bash call failed because of the agent’s mistake. The previous rules sometimes misjudged which bash calls were real failures, flagging correct tests and empty searches; sometimes missing errors were hidden by piped commands. We now use an LLM judge that looks at each call’s purpose, outcome, and cause of any error, and whether or not the error was fixed within the call. This results in a more accurate assessment of whether a bash call failed.
• Scoring. Previously, the score measured how many consecutive turns it took a model to recover after a bash error. This only looked at sessions where something went wrong, so a model with flawless bash use wasn’t being credited. It also treated any successful call after a failure as a "recovery," even though sometimes the subsequent call was not directly related to the original failure. The new score measures bash failures per call, so a session with no bash failures now counts in a model’s favor.

Together, these changes mean the score better reflects how reliably a model uses the command line. The updates are reflected on the new Bash Reliability signal leaderboard. Many models have shifted, so we’re sharing this update because we want the community and our customers to understand exactly how and why it’s changing, and how it shows up on the leaderboard. Backfilled scores under this new methodology will be available in our public HuggingFace datasets soon. The full technical breakdown of the new definition is available on the Agent Arena signal definitions page.

We’ve updated how the Steerability score is calculated on the Agent Arena signal leaderboard. Previously, it only measured whether a model fixed a mistake right after being corrected by a user. This meant that the signal rewarded corrected turns, but it didn’t reward a model for needing fewer corrections in the first place. That made the signal not as accurate as it could be: a model that made more mistakes had more chances to "pass," while a model that got things right the first time earned no credit.

The newest frontier models require corrections less often than other models, but the old score didn’t capture that advantage. The updated methodology now evaluates every assistant turn in a conversation we can judge, so getting things right the first time counts. It also continues to weigh repeated corrections more heavily than one-off corrections that are resolved immediately.

We're sharing this update because we want the community and our customers to understand exactly how and why it's changing, not just see scores move. Backfilled scores under the new methodology will be available in our public HuggingFace datasets soon.

Full technical breakdown of the new definition is available on the signal definitions page.

Leaderboard updates

• ideogram-4.5 has been added to the Image Edit leaderboard.
• gpt-6.1-sol-max has been added to the Code Arena: WebDev leaderboard.
• gemini-4-argon-high has been added to Code Arena: WebDev, Agent Arena, and Text Arena leaderboards.
• hidream-o1-video-1.0 has been added to the Image-to-Video Arena leaderboard.