# The weights didn't change. The score did.

> Anthropic revised a prior model's benchmark score in May 2026; OpenAI nearly tripled another's two months later — both without touching the weights, only the harness. What that says about self-run evals.

Published: 2026-07-30
Tags: benchmarks, evals, ai
Canonical: https://aaronchartier.com/blog/the-weights-didnt-change-the-score-did

---
On May 28, Anthropic [announced Claude Opus 4.8](https://www.anthropic.com/news/claude-opus-4-8). In a footnote, it revised the OSWorld-Verified score it had published for Opus 4.7 — the *previous* model — up to 82.3%. Nothing about Opus 4.7 changed. Anthropic changed how it runs the evaluation and applied the new number retroactively. It said the update would "more accurately reflect the model's performance in the real world."

Two months later, on July 29, OpenAI published its own case study on the opposite move. GPT-5.6 Sol scored 13.3% on the public ARC-AGI-3 task set under the benchmark's standard harness. Enable two settings in OpenAI's own Responses API — retained reasoning across turns instead of discarding it after every action, and compaction instead of truncating context at 175,000 characters. The same weights then hit 38.3%, using a sixth of the output tokens. No fine-tuning, no larger budget, just a different way of passing state between the model and the game. OpenAI was explicit about the cause: the standard harness discarded the model's private reasoning after every move, forcing it to reconstruct the game's rules from scratch on each turn.

Two labs, two months apart, both moved a headline number without touching a single weight. One revised how an old score was computed; the other revised how a live run was configured. Two months rules out coincidence. A benchmark score is a property of a model plus a harness plus a set of runtime flags, and the party running the eval usually controls all three.

ARC Prize administers a separate verified leaderboard for ARC-AGI-3, run under conditions it fixes itself rather than whatever a lab's own team configures. That board still has Sol around 7.8%. Claude Opus 5 sits at roughly 30.2%. OpenAI's 38.3% is a real number — it measures a real capability gap between naive and stateful agent scaffolding — but it doesn't rank against anyone else's, because nobody else's 30.2% was produced under that scaffolding.

This has happened before with a cruder mechanism. In April 2025, Meta submitted "Llama 4 Maverick-03-26-Experimental" to LMArena, a chat-preference leaderboard where humans vote blind between two anonymous responses. The experimental build, tuned for the kind of long, emoji-heavy answers that win human votes, landed at #2 overall. The model Meta actually shipped to developers had none of that tuning, and once people benchmarked it directly, it fell outside the top 30. LMArena's rule was already "don't submit a variant you won't ship" — Meta had just found the gap between what the rule said and what it could verify before the fact. Different failure, same shape: whoever controls what gets tested controls the score, and the discrepancy is only visible once someone tests the thing that actually ships.

The honest complication is that neither Anthropic nor OpenAI did anything like what Meta did. Both disclosed the mechanism in public, in detail, with the old and new numbers side by side. That's closer to a vendor telling you its own thermometer read two degrees warm and handing over the correction factor than to hiding a bait-and-switch. But the correction factor is authored by the party with the largest interest in which direction it points, and there's no obligation to publish every version of the harness that made the model look worse.

The same disease shows up one layer down, in the judges rather than the tests. A July 9 paper auditing LLM-as-judge setups directly found that swapping an evaluator model changes verdicts in ways that don't track evaluator quality. Only one judge upgrade in their whole test matrix produced a statistically significant accuracy gain. [Stacking five judges into a majority-vote jury](https://arxiv.org/abs/2607.08535) moved accuracy from 0.463 to just 0.482, because correlated judges make correlated mistakes. Position-flip rates — how often a judge changes its verdict when you swap the order of the two answers it's comparing — fell from 32% to roughly 12–15% as judges got stronger, but never went away. The referee is now a variable too, and nobody's default harness locks it down.

Self-reported evals aren't fraud. They're closer to a claim than a measurement, and the distinction is whether a party with no stake in the outcome can rerun the exact configuration and get the exact number. A benchmark that adjusts its own scaffolding to match whichever harness produces the best headline number stops measuring model capability. It starts measuring whoever last changed the scaffolding. The benchmarks still worth citing a year from now will be the boring ones: frozen harness, frozen task set, run by someone who doesn't ship a competing model. Everything else is a vendor telling you how well it did on its own test, graded by itself, with the grading rubric attached as a footnote.