Brief, mostly signal

Frozen embeddings with logistic regression beat a local LLM on classification

A vendor blog reports 94.3% accuracy on banking77 at 118 ms on CPU, versus 68% for a zero-shot 14B model. The test used only 100 samples, so treat the gap with care.

Read the source at jeffyclassify.com

What the research found

Vendor blog (Jeffy) claims frozen BGE embeddings plus logistic regression hit 94.3% on banking77 (77 classes) at 118ms on CPU, versus 68% for the best local LLM (phi4 14B, zero-shot, 15.4s). The benchmark is a 100-sample test with a trained classifier against zero-shot prompting, so the gap mostly shows that supervised training beats zero-shot label prompting, not that embeddings beat LLMs in general.

Claims and how well the evidence supports them

Jeffy (BGE embeddings plus logistic regression) scored 94.3% accuracy on banking77 on 100 held-out samples, versus 68% for phi4 14B via Ollama zero-shot 96%
Jeffy latency was 118ms versus 15.4s for phi4 14B on a MacBook Air M2 16GB 98%
The LLM comparison used zero-shot prompting with 77 labels, while the Jeffy classifier was trained on the banking77 training set 93%
Install is pip install jeffy-classify and the embedding model runs locally via ONNX on CPU 98%

Evidence

jeffyclassify.com https://jeffyclassify.com/blog/jeffy-vs-llm-classification

Actionability rated low.