Essay argues most agent eval writing is low quality

What the research found

Opinion essay by Amol Kapoor arguing most agent eval writing is low quality, especially vibe-coded evals that lock in older model behavior; it is a commentary piece with no release or version data.

Claims and how well the evidence supports them

1. An eval only measures whether an AI is good at that specific eval, not general capability 98%
2. Vibe-coded evals that restate what the agent already knows can lock in behavior of older model generations 98%
3. A bad eval is worse than no eval because it locks in practices that optimize for the wrong thing 98%

Evidence

• twitter.com https://twitter.com/noriagent/status/2113338229470441579

Actionability rated low.