Benchmark evaluation depends on difficulty distribution and coverage gaps

This paper formalizes benchmarks as empirical difficulty measures, identifying two failure modes—measure mismatch and coverage gaps—that distort model comparisons and progress monitoring. It introduces metrics to diagnose these issues and a rewriting method, BenchFill, to reduce coverage gaps effectively.

Based on the paper Your Benchmark Is an Empirical Measure Over Difficulty by Yifan Sun, Naicheng Yu, Jiawen Gong, Jingyan Shen and 1 other

Benchmarks as difficulty distributions reveal evaluation pitfalls

Benchmarks are commonly used to compare models and track training progress by averaging model performance over a set of test items. However, this averaging implicitly weights model performance by the distribution of item difficulties in the benchmark, which may differ from the intended difficulty distribution relevant to the application.

If the benchmark overrepresents easy items where models perform similarly, it may fail to distinguish models that differ on harder items, causing misleading leaderboard rankings. Conversely, if the benchmark lacks items in certain difficulty ranges, localized model improvements in those ranges may not affect the aggregate score, obscuring real progress.

This motivates viewing a benchmark not just as a collection of items but as an empirical measure over a continuous difficulty axis. This perspective formalizes two key failure modes: measure mismatch, where the benchmark difficulty distribution differs from the target distribution, and coverage gaps, where intervals along the difficulty axis have no benchmark items, limiting resolution of progress monitoring.

You have read the opening. The rest is free with a Bezen account, and no card is needed.

Create a free account Log in

In the rest of this paper

How the method works, step by step, with a flow chart
Three key formulas from the paper, explained symbol by symbol
Three figures from the paper, and how to read them
How it was tested, what was found, and where it falls short

Benchmarks as difficulty distributions reveal evaluation pitfalls

Sign in to read the whole paper