Brief, mostly facts

Outcommon tests 12 agent setups on a warehouse counting task

The Munich developer reports none of the 12 setups passed the code-graded task, and 37 percent of answers were wrong without a flag. The numbers are self-reported and unaudited.

Read the source at reddit.com

What the research found

A Munich developer's Outcommon project reports that 12 agent setups over 31 self-run tests passed none of the code-graded warehouse sack counting task, with 37% of answers wrong without a flag; these are self-reported, unaudited numbers.

Claims and how well the evidence supports them

12 agent setups were tested over 31 runs and none passed the warehouse sack counting task 98%
In 30 runs 224 of 600 answers (37%) were wrong without a please check mark 99%
Outcommon grades answers with code, not an AI judge 99%
Outcommon is free to try with a Google sign-up 99%

Evidence

reddit.com https://reddit.com/r/ClaudeAI/comments/1xchn9e/new_models_score_great_on_paper_then_fail_my_own
outcommon.org https://outcommon.org/results

Actionability rated low.