Brief, mostly facts
SciPivot benchmarks judgment of research-agent plans
A 248-case benchmark finds frontier agents often prefer flawed research plans. Four of five models score below chance, and the dataset is on Hugging Face.
What the research found
SciPivot is a 248-case benchmark showing frontier LLM agents often prefer flawed research plans over human corrections, with four of five models below chance and best accuracy 63.5% versus 73.4% for fresh researchers; dataset is on Hugging Face.
Claims and how well the evidence supports them
- SciPivot is a 248-case benchmark built from human corrections of LLM agent research plans 95%
- Four of five frontier models perform below chance on selected cases 98%
- The highest project-condition accuracy is 63.5% versus 73.4% for fresh researchers 99%
- At medium reasoning effort models prefer the original LLM plan over human-corrected alternatives 98%
Evidence
- huggingface.co https://huggingface.co/papers/2610.21082
- huggingface.co https://huggingface.co/datasets/sun1245/SciPIvot
Actionability rated low.