Brief, mostly facts
AgentVQA benchmarks visual agents on 1000 manually selected examples
The best of 22 vision-language models scores 58.5 percent. Grounding and state tracking are the main failure modes, and the dataset is on Hugging Face.
What the research found
AgentVQA is an offline benchmark of 1000 manually selected agent examples from 23 source datasets across web, robotics, egocentric video, games and spatial planning; the best of 22 VLMs scores 58.5%, with grounding and state tracking as main failure modes. Dataset is on Hugging Face.
Claims and how well the evidence supports them
- AgentVQA has 1000 manually selected agent examples from 23 source datasets 98%
- Best of 22 evaluated models reaches 58.5% accuracy 99%
- Covers five agent domains: web, robotics, egocentric video, games, spatial planning 99%
- Published on arXiv 2610.31494 on Oct 22, 2026 41%
Evidence
- huggingface.co https://huggingface.co/papers/2610.31494
- arxiv.org https://arxiv.org/abs/2610.31494
Actionability rated medium.