Scale Labs Release Notes

Enterprises everywhere are putting AI agents to work on business-critical workflows. However, measuring whether those agents are any good is harder than it looks. Today's benchmarks are capability tests: they score how often a model gets the right answer working alone, with no people involved. That tells you what it can do. It doesn't tell you whether it's ready to deploy, which is a different question. Big organizations now need to know whether an agent can hit the reliability a workflow requires, at a cost they can plan for, and how much human review it takes to get there.

Say an agent scores 80% on one of those capability tests. That number alone tells you almost nothing about whether it's ready to ship. What matters is the other 20%: whether the agent can flag its own weak answers, whether a person catches the ones it misses, and what that human review costs.

INTRODUCING READY

Today we're introducing READY (Reliable Enterprise Agent Deployment), a suite of industry-specific benchmarks that measure agents the way enterprises actually use them: working alongside people, inside real workflows. For any given workflow, READY Benchmarks measure three things together:

• Reliability: How often the agent and its human operators get the work right.
• Human Oversight: How much human involvement is needed to reach that level of reliability.
• Cost: What it takes to run the agent and the human oversight together.

Measured this way, the result isn't a score. It's a deployment profile: this agent, in this workflow, with this much human oversight, reaches this level of reliability at this cost. Workflows differ enormously from one industry to the next, and so do the roles people play in checking the work. That’s why we’ll release more industry-specific benchmarks under READY over the coming months.

HOW IT WORKS

How READY Works: 3 stages to a deployment profile

READY starts with an enterprise's workflow.

The enterprise defines what counts as correct work and how reliable the system needs to be, and READY finds the most cost-effective way to get there. Every system runs through three phases:

1. Workflow evaluation: The agent works through real cases from the workflow. READY records the whole investigation, including the tools it called and the evidence it pulled, and scores each case against that workflow's own standard for correct work. It also records how confident the agent was in each answer.
2. Policy optimization: Given a reliability target, READY finds the most efficient way to split the work between the agent and the human (i.e., oversight policy). The agent keeps the cases it is most confident in, and the rest go to a human reviewer. READY searches across possible oversight policies and picks the one that meets the target at the lowest cost.
3. Independent validation: READY freezes the chosen split and re-tests it on cases it never saw. The result is certified only when it clears the reliability target with statistical confidence, under the stated assumptions.

RESEARCH FINDINGS

We ran agentic systems through READY, and three things stood out that a capability score would have missed:

• Identical benchmark scores can have opposite deployment verdicts. Two systems separated by only 0.3 points in autonomous accuracy (72.8% vs. 72.5%) require 39.2% versus 29.6% human review, respectively, to qualify at the same 76% reliability target.
• Accuracy, and knowing when to escalate, are different skills. Across the 16 systems, accuracy and routing quality were essentially uncorrelated (Pearson ρ = −0.12). Qwen-3.7-Plus is the clearest case, pairing one of the lowest accuracies with the best routing signal of any system tested.
• Reliability tops out at the human reviewer's own accuracy. Under an assumed 90% human-review success rate, no policy exceeds 90% reliability, and every system is driven to full human review by a target of about 85%.

WHAT THIS MEANS FOR ENTERPRISE AI

A READY deployment profile is deliberately specific: this agent, in this workflow, with this oversight model, under these assumptions. It doesn't transfer to a different workflow, and it isn't meant to. Deployment decisions are always situational, and a single portable score is exactly what has been failing the teams trying to make one.

The testbed is open, and the method works the same way for any workflow. We're launching with two healthcare benchmarks, CliniCARE-Bench and PSEBench, with other industries, such as financial services, coming soon.

Building those out takes people who know the work. We're looking for domain experts to write realistic enterprise tasks and researchers to build evaluation environments. If that's you, sign up here.

Resources:

• Paper (arXiv): https://arxiv.org/pdf/2609.02095
• Companion work:
  • CliniCARE-Bench: labs.scale.com/blog/clinicare-bench
  • PSEBench: https://labs.scale.com/papers/psebench
• Built on Inspect AI

Since the initial release, we have continued refining MultiChallenge, our benchmark for evaluating multi-turn conversational performance across four core capabilities: instruction retention, inference memory, self-coherence, and reliable version editing. This update focuses on improving evaluation reliability and reducing subjectivity in the dataset.

We are introducing three core changes:

• A more aligned judge model: upgraded the LLM-as-judge to Gemini 2.5 Pro, which better aligns with human ratings
• Dataset refinements: updated ~54 tasks to reduce ambiguity and improve evaluation consistency
• Refreshed leaderboard results: re-ranked based on the latest frontier models

For a full description of the benchmark design, challenge definitions, and evaluation methodology, see the MultiChallenge leaderboard page.

A BETTER JUDGE MEANS MORE RELIABLE EVALUATION

MultiChallenge uses an automated evaluation system based on LLM-as-judge with instance-level rubrics. For each task, human annotators define a binary rubric question that determines whether a model response passes or fails. Because these rubric questions depend only on the final model output, they enable scalable automated evaluation while maintaining strong agreement with human raters.

In this update, we upgraded the judge model to Gemini 2.5 Pro to further improve alignment between automated judgments and expert human evaluation. With this change, judge–human agreement improved by more than 5 percentage points compared with the previous evaluation pipeline.

DATASET REFINEMENT: REDUCING TASK SUBJECTIVITY

Evaluating multi-turn conversations is inherently challenging. Because natural dialogue often allows multiple reasonable responses, small ambiguities in task wording or evaluation criteria can introduce subjectivity into benchmark results. As part of this update, we reviewed all tasks and revised 54 tasks in the MultiChallenge dataset to reduce these sources of ambiguity. The revisions focused on two areas:

• Tightening evaluation rubrics by refining the instance-level binary rubric questions used in automated evaluation
• Removing edge cases where multiple responses could plausibly be judged correct

These changes do not alter the fundamental difficulty of the benchmark. Instead, they make evaluation more precise and less sensitive to interpretation, increasing confidence that model successes and failures reflect genuine differences in conversational reasoning.

UPDATED RESULTS

With the dataset refinements and upgraded evaluation pipeline in place, we re-ran MultiChallenge across the latest frontier models to establish an updated leaderboard.

Current top rankings:

• gemini-3-pro-preview: 65.67
• gpt-5.1-2025-11-13-thinking: 63.41
• gpt-5-thinking: 63.19

Check out the full results on the updated leaderboard.

TRENDS OVER THE LAST YEAR

After releasing the MultiChallenge dataset early last year, we tracked model progress across the benchmark. The per-axis breakdown shows where models improve and where challenges persist. The figure below shows per-axis accuracy across all evaluated models.

Models have made real gains on Inference Memory and Instruction Retention, these two axes show the most improvement as models scale, while Self Coherence and Reliable Version Editing remain stubbornly difficult for current models.

Inference Memory shows the widest spread and the highest top scores. Gemini 3 Pro leads at 81.96%, with multiple other models scoring above 70%. Reliable Version Editing is the most difficult axis for the models. To put the RVE gap in perspective: the average RVE score (~38.5) falls below the floor of what any top-10 model achieves on the other three axes. It seems that editing previous versions of its own responses appears to be a qualitatively harder challenge for models than following instructions or recalling information.

One of the most valuable uses of a stable benchmark is tracking generational progress within model families. Below figure shows how models improved across generations.

Gemini shows the steepest climb of any model family, nearly doubling from earliest to latest generation. Gemini-3-pro is now the best performing model on the dataset at 65.67. GPT has also made dramatic gains from earlier generations but performance appears to plateau across recent models: the leap from GPT 5 Thinking (63.19) to GPT 5.1 Thinking (63.41) is just +0.22 points. Claude models have shown the least improvement overall. Claude 3.7 Sonnet was the strongest model at last year's launch, but gains on multi-turn capabilities have been modest, Claude Opus 4.5 Thinking reaches only 58.97.

A striking pattern emerges at the top of the leaderboard. Gemini 3 Pro (65.67), GPT 5.1 Thinking (63.41), and GPT 5 Thinking (63.19) seems to land within a 2.5-point performance gap. It would be interesting to see whether 66 represents a soft ceiling for current approaches or a barrier the next generation will break through.

Another interesting observation is that open-source models remain far behind the frontier. Deepseek v3.1 (46.10), GPT OSS 120B (45.34), and Qwen 235b (41.22) all trail the top closed models by a wide margin, with the gap largest on Reliable Version Editing and Inference Memory.

WHAT THE RESULTS TELL US

Despite the impressive capabilities of today’s leading models, conducting natural multi-turn conversations with human users remains challenging. Success requires models to retain instructions across turns, recall earlier user information, revise evolving artifacts, and remain coherent with their own prior responses. The updated MultiChallenge benchmark provides a more reliable way to measure these capabilities.

Several trends are worth watching going forward. First, Reliable Version Editing remains an open problem: the best score of 57.50 means even the top model fails on iterative editing tasks more than four times out of ten. Second, top models appear to converge around the 65-point mark, raising the question of whether current training approaches are approaching diminishing returns on multi-turn tasks or whether new techniques will unlock the next step change. Third, open-source models need to close a 15-20 point gap on multi-turn conversations before they can compete with closed-source frontier models.