Artificial Analysis has updated its Intelligence Index to version 5, which the company calls a major upgrade toward a more accurate ranking. The most visible change concerns the distance between open-weight and closed models: the gap grows from 3 to 12 points. The new leader among open-weight models is Kimi K3.1 Pro, at 46 points, but its weights have not yet been released. Among weights already published, DeepSeek V4.1 Flash 1017 (max) and GLM-5.3 (max) lead with 32 points, 26 behind the first, and remain below Claude Haiku 5.5 (max).
What changes in the methodology
The structural novelty is the weight of private tests. According to Artificial Analysis, 60% of the index is based on evaluations with private test sets or held-out answers, meaning answers that are not published. It is the first time most of the score depends on data nobody can consult. As a result, the scores cannot be reproduced independently: anyone who wants to check them must trust the company that calculates them. v5 introduces two new benchmarks and three upgrades over the previous version.
Image: Artificial Analysis — https://artificialanalysis.ai/changelog
Questions about accuracy
Some users on the r/LocalLLaMA community have already questioned the ranking's accuracy, comparing scores with their own experience. The most cited case is a model that ranks above another which, in daily use, is judged better. This is not a scientific test, but it is the kind of signal that makes one ask how well an aggregate index reflects real tasks.
The index's background
v5 follows v4.2, published on September 4 as an interim update with more complex tasks and more private test sets to prevent gaming. Already in that version the company said it was accelerating elements of v5. In recent months the index has tracked the race between Chinese and American models: DeepSeek V4.1 Flash, Qwen3.8 Max and GLM-5.3 have alternated at the top of the open-weight ranking. For anyone comparing models, the new scale makes it harder to tell whether a higher score signals a better model or simply a different test from the previous ones.
Image: benjaminhan.net — https://benjaminhan.net/posts/20260620-intelligence-benchmarking/index.html
Who is affected
Developers and companies that choose a model based on the index now face a change of reference: v5 scores cannot be compared directly with earlier versions. For those wanting to use open models, the 12-point gap is the most concrete figure: it shows how far releasable weights trail the strongest closed models by this measure.
The next expected step is the release of Kimi K3.1 Pro's weights, which today tops the index among open models but cannot yet be downloaded.
No comments yet. Be the first!