Artificial Analysis has rebuilt its Intelligence Index, the most prominent independent score for frontier AI models, and the fifth version lines up eight closed models above the best open-weight one. Nathan Lambert, who writes the Interconnects newsletter, published an essay on the change and argues it is mostly a sensible move that leaves a hole in the picture.
Version 5 splits the score into four equal parts of 25 percent each. Knowledge work, coding, science, and a general category all count the same. Coding leans on FrontierCode 2.0 at 15 percent and Terminal-Bench 4.0 at 10 percent, while the science block rests on Terminal-Bench-Science and a hard subset of CritPT v2. Artificial Analysis describes the mix as reflecting real-world usage.
Lambert finds the resulting ranking reasonable at first glance and a little strained on closer inspection, since GPT 6.1 Terra edges out Claude Fable 5.1. His read is that Artificial Analysis is tuning the index toward a general-purpose agentic collaborator, the kind of model that handles open-ended work inside whatever coding harness you prefer. That favors large closed models. Kimi K3.1 Pro, the strongest open-weight entrant he mentions, sits below all eight of them.
Lambert calls the choice defensible but warns about the long game. If workflows built on open models grow into a bigger share of the economy, an index tilted toward the frontier could end up measuring the wrong engine. He also admits the old index flattered open models, and points to a chart from the launch of Kimi K3.1 Pro: the models are useful and strong, yet clearly behind the best OpenAI and Anthropic systems on flexibility and reliability.
He does not expect the gap to last. Open labs have closed these gaps again and again, he writes, and he predicts that the next round of releases, which he guesses could be Kimi 3.5 or GLM 6, in two to three months, will make substantial jumps on this index. The case for open models, in his telling, was never about beating the frontier. It rests on pushing high-demand inference down to commodity hardware prices and on letting companies own an intelligence tuned to their internal data.
The sharper dispute is over secrecy. Artificial Analysis moved to private benchmarks, with hidden prompts and test environments, and Lambert reads that as a response to open labs tuning their models to the public tests. He thinks the best open models do some of that and the worst do a lot. Still, he would rather see transparency, because a hidden test means total trust in whoever built it and removes the community that regularly finds bugs in public ones.
Incentives run through the whole story. Lambert says Anthropic and OpenAI stopped watching them closely long ago, because their effort goes to internal uses such as automating AI research and to customers spending billions. Small and medium labs have the opposite problem: their brand is catching up to the frontier, so a weak showing is existential for them.
He is open about whom he trusts. Artificial Analysis earns money from labs that pay it to verify results, and Arena funds itself with pre-release testing for labs. Lambert says that is fine, and that he trusts both, though not as much as he trusts non-profit groups, university researchers, and public bodies such as CAISI. Those groups, he notes, are not the ones running the frontier evaluations today.
His proposed fix is more indices with different biases, such as CursorBench for coding. Vals AI and Intelligence.ai are growing, but Lambert writes that attention still piles onto Artificial Analysis.
A team choosing between an open and a closed model this quarter gets more from a score built on its own tasks than from a 25 percent science weighting someone else picked.
Reported by Interconnects (Nathan Lambert) in the essay “Artificial Analysis v5 and the benchmarker’s dilemma”; the source file carries no publication date.
