Friday, 13 November 2026 · 6 min read
SO-Arena benchmarks System One models' decision capabilities and reliability across domains
SO-Arena evaluates 21 System One models on 2,919 tasks spanning 11 domains, measuring capability spectrum with difficulty tiers and decision reliability via option order stability and missing-answer recognition, revealing that strong performance does not guarantee reliable decisions.
Based on the paper SO-Arena: Benchmarking the Capability Spectrum and Reliability of System One Models by Lichen Li, Ziping Dong, Sicheng Feng, Qi Li and 1 other
Challenges in evaluating System One decision models comprehensively
System One models make fast, general-purpose decisions by predicting over predefined outcomes in a single forward pass, avoiding lengthy token generation typical of System Two reasoning. They are used in tasks like routing, tool selection, and verification where speed and structured outputs are critical.
Existing benchmarks often report aggregate accuracy but fail to capture the full capability spectrum across diverse domains or the reliability of decisions under varying conditions, such as option order changes or missing correct answers.
Understanding both the breadth of capabilities and the reliability of decisions is essential for deploying System One models in real-world workflows, motivating a benchmark that evaluates these complementary aspects in detail.
Sign in to read the whole paper
You have read the opening. The rest is free with a Bezen account, and no card is needed.
In the rest of this paper
- How the method works, step by step, with a flow chart
- Three key formulas from the paper, explained symbol by symbol
- Three figures from the paper, and how to read them
- How it was tested, what was found, and where it falls short