Numbers the RewardBench 2 leaderboard doesn't show, computed from the per-prompt scores each model uploaded to allenai/reward-bench-2-results. Covers the 168 sequence-classifier reward models with usable score files. Generative judges upload verdicts but no scores, so they aren't here.
Click a header to sort; scroll sideways for all columns. Matching rows are highlighted in the charts. Rows in grey are left out of the medians: two models whose uploaded scores contradict their leaderboard entry (see below) and four debugging runs trained on 50–100% flipped labels. Column definitions are at the bottom.
Each model's score here is its average accuracy over the five non-Ties subsets, which is the leaderboard formula minus Ties. The line through each dot is the range you would get if the benchmark had been built from a different random sample of the same prompts: the prompts were resampled 500 times within each subset, and every model was scored on the same resamples, so two models can be compared on identical draws.
The top model is clearly ahead of everyone else. After that, most neighbours in the top 30 are within noise of each other: only 6 of the 29 adjacent pairs are reliably ordered (marked ▸). A difference of one or two places on the leaderboard doesn't mean anything.
Every RewardBench 2 prompt has four candidate answers, one of them correct. Sometimes the correct one happens to be the longest. A model that partly scores by length will do better on those prompts than on the ones where the right answer is shorter. The bar is that difference in accuracy. It is measured inside each subset and then averaged, because subsets differ in how often the correct answer is longest (12% on Focus, 29% on Factuality) and pooling them would mix that in.
The typical model is about 10 points more accurate when the correct answer is also the longest (median +10.5, positive for 154 of 168 models). The effect is biggest on Factuality (+16) and Math (+14), smallest on Precise IF (+4.5). This is an association, not proof that length is being rewarded directly: the longest answer can be correct for reasons that also make it long, like showing its working. But a model that ignored length would sit near zero, and most don't.
A reward model gives each answer a number. For a model trained with the Bradley-Terry loss, the difference between two numbers is meant to be a log-odds: a margin of 2 says the first answer is better with about 88% probability. In practice each model's margins come out stretched or squashed by some factor, and the factor differs from model to model. T is that factor, fitted so that the model's implied probabilities match how often it is actually right (fit on half the prompts, checked on the other half).
T = 2.5 means the raw margins are 2.5 times too big: the model sounds more sure than it is. T = 0.7 is the opposite. Half the models fall between 0.73 and 1.68, and the 10th–90th percentile is 0.59 to 3.0, so two models with the same accuracy can give margins that differ by a factor of five. None of this affects ranking. It matters when the score is used as a number: as a reward in RL, or to set a threshold for best-of-n sampling. Divide the model's scores by its T first.
After that correction, every model is about equally well calibrated. The usual calibration metric (ECE, in the table) is below 0.05 for the whole top 30 and the differences are smaller than its sampling noise at this many prompts, so there is no point ranking models on it.
Per subset, median over models, after the T correction. "Correct is longest" is the share of prompts where the correct answer is the longest candidate, which is why the length gap is measured within subset. "Conf − acc" is how much the model's own probability on its top pick exceeds its actual hit rate. On Precise IF models get 37% right but act like 45%. On Focus, Factuality and Safety they are more right than they think.
| Subset | Prompts | Correct is longest | Median accuracy | Median confidence | Conf − acc | Median ECE | Median length gap |
|---|
Each model has two files on the Hub: a summary the leaderboard reads, and the per-prompt scores used here. For these two models the per-prompt file gives lower accuracy than the summary on every subset, by 5 to 42 points. Each file is consistent with itself, so they come from different runs and one of them is wrong. Both are left out of everything above.