Act on this, mostly facts

Mica v0.5 4B decision model runs in 19.8 ms per decision

What to do

Try the Apache-2.0 Mica v0.5 model, using the INT8 build on NVIDIA or GGUF on AMD, Apple and CPU.

Why

It scores 84 of 111 on JevBench Hard, and the speed figure was measured on an RTX 6000 Ada.

What the research found

Mica v0.5, a 4B Apache-2.0 decision model, reports 19.8 ms per decision on an RTX 6000 Ada and 84/111 on JevBench Hard; INT8 is fastest on NVIDIA, GGUF builds cover AMD, Apple and CPU.

Claims and how well the evidence supports them

19.8 ms per decision at 256 tokens on a single RTX 6000 Ada 99%
84/111 on JevBench Hard, best reported for a 4B model 95%
Apache-2.0 license, runs on 8 GB cards and up 99%
INT8 build about 2 to 2.7x faster than Q8_0 GGUF on NVIDIA 98%

Evidence

reddit.com https://reddit.com/r/LocalLLaMA/comments/1xbw3oy/mica_v05_the_smartest_4b_decision_model_is_also
huggingface.co https://huggingface.co/sky7350/Mica-v0.5-4B
github.com https://github.com/akivet/Mica

Actionability rated medium.