Exa's ATLAS benchmark tests agents on search-heavy tasks

What the research found

Exa previews ATLAS, a 547-task agentic search benchmark with verified golden answers; no agent under $1 per task reaches row F1 above 0.5, and even the most expensive agents miss about a third of golden results.

Claims and how well the evidence supports them

1. ATLAS benchmark has 547 tasks 99%
2. No agent costing under $1 per task achieved row F1 above 0.5 98%
3. Even the most expensive search agents miss about one third of golden results 98%
4. Exa scores vary 16 percent across search backends with harness fixed 83%

Evidence

• exa.ai https://exa.ai/blog/atlas-benchmark

Actionability rated low.