DecisionPivot shows aggregate accuracy hides decision-model failures

What the research found

DecisionPivot is a diagnostic benchmark (48 tasks, 40,033 items) for LLM decision models, showing aggregate accuracy hides contrast-group failures, calibration gaps and reasoning-effort effects. Research paper only; no release or code availability is confirmed in the cached text.

Claims and how well the evidence supports them

1. DecisionPivot comprises 48 tasks and 40,033 items across seven decision roles 99%
2. Core track has 7,800 items from 13 generated task families; reference track has 32,233 items from 35 public test sets 99%
3. Perplexity Decider ranks positive above negative in 92.3% of pairs but answers both correctly in only 74.1% 99%
4. GLM-5.3 reaches 93.5% contrast-group accuracy at higher reasoning versus 81.5% at lowest effort 99%

Evidence

• huggingface.co https://huggingface.co/papers/2610.28522
• arxiv.org https://arxiv.org/abs/2610.28522

Actionability rated low.