Brief, mostly facts
BizBench shows agents break constraints even when goals are met
The 50-task benchmark uses 789 docs and 601 tools. Frontier agents often break intermediate constraints on the way to a goal.
What the research found
BizBench is a 50-task multi-turn benchmark (789 docs, 601 tools) from a payments company showing frontier agents often break intermediate constraints even when goals are met.
Claims and how well the evidence supports them
- BizBench has 50 tasks, 789 knowledge documents and 601 business tools 99%
- Claude Opus 5 and GPT-6 Sol achieve strict success of 64.0% and 61.3% 99%
- Qwen3.8-27B reaches 42.0% goal attainment but 32.7% strict success 99%
- DeepSeek-V4.1-Flash averages 191.3 tool calls per run, about 70% more than reference 98%
Evidence
- huggingface.co https://huggingface.co/papers/2610.20424
- huggingface.co https://huggingface.co/datasets/ant-intl/BizAgentBench
Actionability rated low.