A decision-model controller raised agent cost 19% without better results

What the research found

Pre-specified study on Terminal-Bench 2.1 finds the Jev typed decision-model controller raised agent cost per attempt 19% without improving pass rate (73.0% both arms), was poorly calibrated, and never chose to abandon a run.

Claims and how well the evidence supports them

1. Jev raised cost per attempt by 19% (geometric-mean ratio 1.19, 95% CI 1.02-1.36) at identical 73.0% pass rate 99%
2. Jev was 22% more expensive than a placebo controller and cheaper than a small LLM controller (ratio 0.81) 98%
3. Jev ranked finish requests better (AUROC 0.78) than rules (0.62) and small LLM (0.60) but was poorly calibrated: mean completion probability 0.50 when 87% of complete runs passed 98%
4. No controller ever chose to abandon a run; study used 21 held-out tasks and 315 trials 99%

Evidence

• huggingface.co https://huggingface.co/papers/2610.24649
• arxiv.org https://arxiv.org/abs/2610.24649

Actionability rated low.