Measured results, reproducible from one command

Most benchmark questions are not worth asking. These are.

Choosing between AI models means running a benchmark: a few thousand questions, put to every model on the shortlist, counted up into a percentage. Twenty models is sixty thousand paid calls, and it has to be repeated every time any vendor ships anything. Most of those questions separate nobody. Standardised testing worked this out decades ago and stopped handing everybody the same fixed paper. This page does the same thing to a language model benchmark, on real recorded answers, and lets you watch it happen.

10 questions rank a shortlist as well as 127 chosen at random

– to rank twelve models against the price of asking all of them everything

1 in 5 of the bank measures nothing and every published benchmark score counts them at full weight

Twelve current models from four vendors and a laptop, 2,816 real questions, and the answers they actually gave, recorded in September 2026 for US$16. The adaptive test runs in your browser against those answers; it can only replay what was asked and paid for, and it never invents a response.

The idea in one table

A benchmark score is a count of correct answers. A count cannot tell these five kinds of question apart, so it treats them as equal evidence about which model to buy.

What a question does	What it tells you about which model to buy	What a percentage does with it
Every model gets it right	Nothing. It separates nobody	Counts it, at full weight
Every model gets it wrong	Nothing. It separates nobody	Counts it, at full weight
Its stored answer key is wrong	Worse than nothing: it penalises the models that were right	Counts it, at full weight
It leaked into training data	Measures memory, not capability	Counts it, at full weight
It cleanly separates stronger models from weaker ones	This is the entire signal you are paying for	Counts it, at full weight

Nobody knows which of a benchmark's questions belong to which row, because results are published as one aggregate and the per-question detail is thrown away. Item response theory, the statistics behind the GRE and every professional licensing exam, measures each question first: how hard it is, and how sharply it separates ability just above its threshold from ability just below. Once each question carries those two numbers, the test can be short, because you can ask the question that tells you the most about this particular model and stop when the answer is decided.

Run the test yourself

Pick two models. The test asks one question at a time: after each answer it updates what it believes about each model, then chooses the next question to be the one that will tell it the most about them at their current level. Beside it, the same two models are asked questions drawn at random from the same suite. Same models, same budget, same recorded answers, and the only difference is which questions get asked.

First modelan alias: the vendor mapping lives in the gateway, not here

Second modelthe closer these two are, the more questions it takes

Speedhow fast the questions arrive on screen

Adaptive: ask the question that tells you the most

the method

0questions asked, each

$0.00spent, at the price these calls cost

–width of the wider interval

Not started.

Random: ask whatever comes up

the baseline

0questions asked, each

$0.00spent, at the price these calls cost

–width of the wider interval

Not started.

The questions the adaptive test chose, in the order it chose them

#	Benchmark	Difficulty	Discrimination	First	Second

Watch the difficulty column. The test starts near the middle of the scale, and as it learns that a model is strong it stops wasting questions the model will certainly get right. A negative discrimination in that table is not a mistake either: the selector treats information as information, and an item whose fitted slope runs backwards still carries some. It is also the signature of a wrong answer key, which is why the repository publishes a list of them rather than quietly dropping them here.

Nothing here is simulated. The answers are the ones the twelve models gave when they were administered the frozen suite in September 2026, one bit per cell, and the dollars are the ones the gateway's ledger recorded for those exact calls. The arithmetic, about two hundred lines, is in irt.js beside this page. The item parameters were fitted to 150 other models from a public evaluation archive, and not one of the twelve models here was in that fit, which is the only version of the claim worth anything: the questions were measured before these models existed.

Two models can also fail to separate, and that is a real answer rather than a failure. It says these two are indistinguishable for what you were willing to spend, so stop paying for evaluations and choose on price, latency or context window instead.

One pair is an anecdote. Here are all sixty-six.

Every pair of the twelve models, both methods, same budget, in your browser. It takes a few seconds and nothing is fetched to do it.

Where it wins, and where it stops winning

One run is an anecdote. This is the whole thing: how closely a short test reproduces the ranking a full benchmark produces, against the number of questions it was allowed, with a bootstrap interval on every point. The dashed line is what the same budget buys spent at random.

The honest part of this chart is the right-hand end. Above roughly two hundred questions adaptive selection stops winning and a plain random sample catches it. That is not a broken estimator, it is two different quantities: the adaptive test converges on the bank's ability scale, while a leaderboard reports the suite average, and even with every question in hand those two rankings agree only to about 0.92. If you want the benchmark's own average, sample at random and score it directly. If you want to know which model is better, ask ten well-chosen questions. The project publishes the ceiling rather than cropping the chart before it.

How many questions does your decision actually need?

This is the step almost everybody skips, and it is where an evaluation budget goes. Before spending anything, ask what size of difference the decision turns on. If your candidates are about a point apart, no affordable test will separate them, and learning that first is itself a result.

Difference to detect in percentage points of benchmark accuracy

Where the model sits weak models and strong ones are not equally easy to measure

How often you want to catch it statistical power, at the usual 5% false-alarm rate

– questions per model, chosen adaptively

Asked of a shortlist of five
At the price this panel's calls cost
Share of the 2,816-question suite

Treat every number here as a floor. It assumes questions are independent evidence and they are not: questions share passages, templates and subjects, which the project measures rather than assumes away. On this bank a hundred mathematics questions carry about six questions' worth of independent evidence.

One question in five is measuring nothing

Every dot is one question from the bank of 20,365. Across: how hard it is. Up: how sharply it separates stronger models from weaker ones. Everything in the shaded band is close to a coin flip for everybody, and everything below zero runs backwards, meaning stronger models get it wrong more often, which is the signature of a wrong answer key.

A negative slope has four possible causes and they are not equally interesting: a wrong answer key, an ambiguous question where the better model sees the ambiguity, a grader marking a correct answer wrong, and genuine inverse scaling. Telling them apart needs the question text and a person, so nothing here is published as mis-keyed. But one fit produces the list, and if you own a benchmark it is where to look first: those questions are free to find and they cost you money on every run.

Three things a single score cannot tell you

The same twelve models, measured three more ways. Each of these is cheap to run and almost nobody publishes them, which is why a two-point drop in a leaderboard gets a headline it has not earned.

What this does not do

It scores right or wrong, and nothing else

No writing quality, no human preference, and no second model acting as a judge. The psychometrics rests on a correct-or-not response, and a judge would add a second, drifting instrument to the measurement.

Questions are not independent, and that is measured

Between 8 and 31 percent of question pairs, depending on the benchmark, are correlated beyond what ability explains. Every interval from a fixed-length benchmark is therefore optimistic, including some of this project's own, and the correction ships with the bank.

Twelve models is twelve models

The live half of this page rests on twelve models spanning 0.54 to 0.94 accuracy across four vendors and a laptop. That settles that the question parameters transfer at all. It is too few to rank methods against each other, and every interval in it is wide and published wide.

It replicates, which is why there are two banks

A result measured on one panel is a result about that panel, so the whole method was run again on 400 open-weight models from a different source with a different harness. The shape replicates and so does the ceiling. Item parameters transfer between the two only after the questions that measure nothing are dropped.