Three threads run through today’s issue: labs grading their own work, cheap downloadable models taking on real jobs, and a few companies letting outsiders look inside.
Nearly every big model claim today comes from the company selling the model. Microsoft, NVIDIA, Krea and Liquid AI all published scores, and not one has been repeated by anyone else.
Meanwhile the money is quietly moving. Vercel says downloadable models carry 71.5% of the work on its gateway for 23% of the spend, and Meta shipped a giant model that fits on two desktop Macs.
The oversight news is better. Anthropic gave a nonprofit staff-level access with the right to publish, and Perplexity released the locked room its agent works in.
Marking Their Own Homework: Scores Nobody Else Has Run
Four launches today arrive with the headline numbers supplied by the company that built the thing. That is not proof they are wrong, only that nobody has had a chance to check.
- Microsoft says its giant AI model beats rivals, on scores only it has run. Microsoft’s in-house lab says its 2.8 trillion-parameter model beats every open rival it tried while using far less computing power. Nobody outside has tested it, and nobody outside can use it yet.
- NVIDIA’s new AI model finishes work 4x faster, by its own measure. The four times faster figure for agent tasks has no named baseline in NVIDIA’s post, so it is unclear what it is four times faster than. The post also names no licence.
- A new AI photo editor claims to change only the part you point at. Krea says its editor takes under 8 seconds and about $0.06 a go, leaving the rest of the picture alone. The proof is Krea’s own comparison, with one attempt per rival.
- Liquid AI’s cheap photo-judging model says yes or no, and how sure it is. Liquid AI says its model beats rivals at factory-style pass or fail calls, at $0.042 per million tokens. Every comparison in the post is the company’s own.
Small Bills, Big Models: Downloadable AI Does the Heavy Lifting
Models you can download are taking a large share of the actual work, and they are getting small enough to run on hardware people already own.
- Downloadable AI models do 71.5% of one platform’s work for 23% of its money. This is Vercel’s own traffic through its AI gateway, so it describes one platform and not the whole market. The pricier closed models still bring in most of the revenue.
- Meta releases a giant AI model that runs on two desktop Macs. Muse Spark Open uses only a small slice of itself at a time, which is why two Mac Studios can hold it. Meta’s post gives no speed, price or benchmark figures, and the licence is limited commercial.
- A free image model small enough for a gaming PC claims to beat bigger ones. Black Forest Labs says its model runs on an 8 GB graphics card and tops larger rivals. The rankings are its own, and free commercial use stops at $5M in revenue.
Inside the Box: Who Gets to Look
Three stories about opening things up, from an outside group with a key to a locked room anyone can now inspect.
- Anthropic lets an outside group inside, and lets it publish what it finds. Six staff from the nonprofit METR now work with employee-level access, and Anthropic cannot veto what they publish. Who pays METR, and what stays off limits, is not clear.
- Are the insides of AI models really tangled? One researcher says no. A Goodfire essay argues the mess in neural networks was misread, so they can be studied directly. The evidence is substantial but not yet peer reviewed, and whether it holds at frontier scale is open.
- Perplexity open-sources the locked room its AI agent works in. Nobody broke out of the virtual machine in 108 attempts, though some attacks got around its network limits. Perplexity says it fixed those holes and that neither method worked a second time.
After the Launch: What Goes Wrong in Practice
Two accounts of building AI that works, one from a training pipeline and one from a product that had to start over.
- Training an AI to use tools goes wrong in ways the AI doesn’t cause. The argument is that the software around a model can quietly corrupt what the model learns. The post describes fixes others use but runs no experiments of its own.
- OpenAI launched Codex badly, then rebuilt it. What changed. The first version took hours to set up and could not be interrupted. The fix was to run on the user’s own computer, and the 40 million user figure counts more than Codex alone.
Today’s Quick Hits
- OpenAI adds six languages to its voice mode, with the API still pending. French, German, Hindi, Mexican Spanish, Korean and Japanese voices arrived in ChatGPT on 15 October. Developers have no date for access.
- Ordinary JavaScript, turned into a neural network with no training. Cheng Lou’s toy project writes a program’s logic straight into a network’s numbers, so the answers match exactly. Chosen parts can then be unfrozen and learned.
- A researcher argues fine-tuning can only change a model in certain places. Yingru Li says changing a little of a model and changing the same little each time are different claims. Where a model can change, he argues, is visible before any training reward arrives.