Hamel Husain compares decision models for LLM eval failure detection

What the research found

Hamel Husain's post compares decision models (Jev 1.13, Perplexity Decider, Liquid d1, Cloudflare Clef Flash, OpenAI Decisions) for LLM eval failure detection on 100 leasing-agent traces, arguing the default 0.50 threshold should be tuned and cost and calibration checked.

Claims and how well the evidence supports them

1. The default decision threshold of 0.50 can leave performance on the table and should be checked against task data 98%
2. The benchmark used 100 traces from an apartment leasing agent to test decision models 99%
3. Tested models include Jev 1.13, Perplexity Decider 1.1 27B, Liquid d1 and d1-ultra, Cloudflare Clef Flash, and OpenAI Decisions (GPT-6 Luna) 99%
4. Decision models are a low-cost alternative to LLM judges for automated evals 98%

Evidence

• hamel.dev https://hamel.dev/blog/posts/decision-model-evals

Actionability rated medium.