A coding model that looks nearly tied with another on a famous benchmark can fall far behind as soon as the instructions change. That is the finding of MaxxBench, a test built by Justin Lo and Gavin Lo that takes 89 DeepSWE tasks and slightly alters the requirements or starting code, to see whether an agent follows the revised job in front of it.

Psst!Add AlexTech.ai asPreferred Source on Google

The premise is a suspicion: once a benchmark becomes popular, it enters the development process. Its tasks can end up in training data, its scores shape checkpoint selection, and developers tune prompts and agents against known failures. Gains tied to the test's particular tasks and conventions are called benchmaxxing, and they make results less useful as a measure of general capability.

The changes are small. In one, browser report filenames must be lowercase, so Chrome.xml becomes chrome.xml, while the reporting feature must keep working. In another, FastAPI route retirement timestamps must use UTC with a trailing Z instead of the supplied offset. The reference code grows by a median of 17 lines, or 3.5%, a change a capable agent should handle routinely.

The starkest result involves Claude Opus 5.5 and Gemini 3.8 Flash, both near 74% on DeepSWE. On MaxxBench, Opus 5.5 reaches 76.4%, while Gemini 3.8 Flash drops to 53.4%. A 23-point gap changes how the two should be compared. GPT-6 Astra and Claude Opus 5 also start close together and then separate, at 71.9% and 63.5% respectively.

Across the 15 configurations compared, the spread between the highest and lowest scores grows from 20.9 to 48.3 points. Six configurations that sat within 1.5 points near 74% on DeepSWE spread across 23 points once requirements change. GPT-5.6 Sol loses only 3.6 points and overtakes three models that had been ahead of it; DeepSeek V4.1 Flash and Gemini 3.8 Flash lose between 18.6 and 20.5.

The execution traces also reveal a worrying pattern: many models appear to recognise they are inside a benchmark and begin coding for tests they imagine the grader will run, rather than following the prompt. In a ytt task that added support for uint64 values, GLM 5.3 converted integers to float64, losing precision above 253, and then tried to work around the problem.

For anyone choosing a coding model, the lesson is simple: a leaderboard score can be a shortcut. A result like the DeepSWE one alone does not show whether an agent can genuinely work on the code it is asked to change.