A viral post says GPT-6 Astra can improve an open model by studying its raw weights, and a new critique on honnibal.dev says the post stitches together experiments that do not work the same way. Matthew Honnibal, whose blog carries the argument, is the second critic of the paper to appear here, after Sean Goedecke’s objections yesterday, and his complaint is different and narrower.
His target is the structure of the post, not the honesty of any single result. He compares it to a stage magician who performs one trick several ways, so that each version seems to rule out the explanation you formed from the last. The audience assumes one method is at work. The post, he says, invites the same assumption.
The abstract says Astra can work out a network’s training purpose “just by looking at the raw weights,” and then describes the Qwen3.5-0.8B result in the next breath. Honnibal’s reading of the post’s own table is that this setting, meaning raw numbers in the prompt, no tools and no running the network, shows up in only three of the experiments. Those networks hold between 17,000 and 172,000 weights. Each test on a real language model relies on Python code.
That includes the headline one. According to Honnibal, the Qwen fix is also allowed to execute the model on as many as 50 batches of practice problems. The post reports that a change to 0.016% of the weights took the MATH-500 score to 48.1% from a starting 7.9%, with IFEval improving threefold. Honnibal does not dispute those numbers. He disputes what they show.
His view is that the Qwen result demonstrates Astra can be prompted to plan and carry out work a human machine-learning engineer might do: write code that inspects or edits weights, test it, iterate. Engineers do this routinely, he notes, and getting a model to do it productively is “not nothing.” Earlier models would have failed these tasks, he adds.
The table also records failures that the abstract leaves out. The same approach did not spot a sycophantic Gemma 3 when Astra could write code but not run the model, and a repeat of the Qwen-style fix on Gemma 3 failed. On the game networks, a footnote says the from-scratch Breakout player beat a PPO baseline trained for 500 million steps by 517 to 25, but lost to it on Seaquest, 53 to 89, and on Asterix, 53 to 55. The abstract compares against a 100-million-step baseline.
Honnibal’s central point is about what the results can prove. One technique that could write a small game network from nothing and also repair a large language model would imply a deep grasp of what weights encode. Separate techniques imply no such thing. A reader who takes the abstract at face value credits Astra with the first, when the experiments support the second.
The stake is practical. Claims like this shape what buyers expect from agents that improve models, and reading numbers cold is a far harder job than running a Python lab with test batches. Two critics landing on the post’s framing within days suggests its weakest point is the packaging, not the engineering.
Anyone planning to benchmark a weight-editing agent against that 48.1% should copy the setup that produced it, Python access and 50 batches of test runs included, rather than the one the abstract describes.
Reported by honnibal.dev, the blog of Matthew Honnibal, in a post titled “The ‘Astra can read and write weights to improve Qwen’ post is misleading” (no publication date shown on the page).
