Most benchmark questions are not worth asking. These are.

The idea in one table

A benchmark score is a count of correct answers. A count cannot tell these five kinds of question apart, so it treats them as equal evidence about which model to buy.

Nobody knows which of a benchmark's questions belong to which row, because results are published as one aggregate and the per-question detail is thrown away. Item response theory, the statistics behind the GRE and every professional licensing exam, measures each question first: how hard it is, and how sharply it separates ability just above its threshold from ability just below. Once each question carries those two numbers, the test can be short, because you can ask the question that tells you the most about this particular model and stop when the answer is decided.

Run the test yourself

Pick two models. The test asks one question at a time: after each answer it updates what it believes about each model, then chooses the next question to be the one that will tell it the most about them at their current level. Beside it, the same two models are asked questions drawn at random from the same suite. Same models, same budget, same recorded answers, and the only difference is which questions get asked.

Nothing here is simulated. The answers are the ones the twelve models gave when they were administered the frozen suite in September 2026, one bit per cell, and the dollars are the ones the gateway's ledger recorded for those exact calls. The arithmetic, about two hundred lines, is in irt.js beside this page. The item parameters were fitted to 150 other models from a public evaluation archive, and not one of the twelve models here was in that fit, which is the only version of the claim worth anything: the questions were measured before these models existed.

Two models can also fail to separate, and that is a real answer rather than a failure. It says these two are indistinguishable for what you were willing to spend, so stop paying for evaluations and choose on price, latency or context window instead.

Try anthropic-sonnet against openai-frontier, which the method cannot settle. Both estimates run to the end of the bank's ability scale and stop there, because the selector asks for questions it believes are a coin flip at that level and these two get about nine in ten of them right. Only 253 of the 2,816 questions are harder than +3, so the suite has run out of questions that could tell them apart, and the tight-looking interval is a wall rather than evidence. The item parameters were fitted on 150 models evaluated between 2023 and 2025; a bank calibrated on that panel measures the top of a 2026 shortlist poorly, which is a limit of this bank rather than of the method, and it is the reason the page draws the edge of the scale instead of leaving you to notice the curve is flattened against it.

One pair is an anecdote. Here are all sixty-six.

Every pair of the twelve models, both methods, same budget, in your browser. It takes a few seconds and nothing is fetched to do it.

Run all 66 pairs

Where it wins, and where it stops winning

One run is an anecdote. This is the whole thing: how closely a short test reproduces the ranking a full benchmark produces, against the number of questions it was allowed, with a bootstrap interval on every point. The dashed line is what the same budget buys spent at random.

The honest part of this chart is the right-hand end. Above roughly two hundred questions adaptive selection stops winning and a plain random sample catches it. That is not a broken estimator, it is two different quantities: the adaptive test converges on the bank's ability scale, while a leaderboard reports the suite average, and even with every question in hand those two rankings agree only to about 0.92. If you want the benchmark's own average, sample at random and score it directly. If you want to know which model is better, ask ten well-chosen questions. The project publishes the ceiling rather than cropping the chart before it.

How many questions does your decision actually need?

This is the step almost everybody skips, and it is where an evaluation budget goes. Before spending anything, ask what size of difference the decision turns on. If your candidates are about a point apart, no affordable test will separate them, and learning that first is itself a result.

One question in five is measuring nothing

Every dot is one question from the bank of 20,365. Across: how hard it is. Up: how sharply it separates stronger models from weaker ones. Everything in the shaded band is close to a coin flip for everybody, and everything below zero runs backwards, meaning stronger models get it wrong more often, which is the signature of a wrong answer key.

A negative slope has four possible causes and they are not equally interesting: a wrong answer key, an ambiguous question where the better model sees the ambiguity, a grader marking a correct answer wrong, and genuine inverse scaling. Telling them apart needs the question text and a person, so nothing here is published as mis-keyed. But one fit produces the list, and if you own a benchmark it is where to look first: those questions are free to find and they cost you money on every run.

Three things a single score cannot tell you

The same twelve models, measured three more ways. Each of these is cheap to run and almost nobody publishes them, which is why a two-point drop in a leaderboard gets a headline it has not earned.

What this does not do