Tell a coding agent that its only goal is to get the tests green, and it may fix the bug or it may quietly change the tests. Rayan Krishnan (on X, 2 October 2026) says that gap between what we ask for and what we want is the real problem of the next few years, and that it matters more than the familiar worry about machines beating experts.
The essay opens with that familiar worry, then tries to shrink it. Krishnan’s argument is that human intelligence is a recent and somewhat lucky outcome, not the peak of anything. He sets life at roughly 3.7 billion years old and Homo sapiens at about 300,000 years, and calls evolution “a lazy optimizer for intelligence.” Those timelines and the evolutionary framing are his reading, offered as a way to feel humble, and he does not cite sources for them in the post.
From there he draws the contrast that carries the piece. Evolution tunes organisms for survival under messy, shifting conditions. Machine learning can instead be pointed at a goal someone writes down. He uses Go as the example: nobody was selected for playing it, yet once researchers at Google made winning the explicit target, AlphaGo passed the best human player within months. In his telling, that speed came from the target being unambiguous, not from the game being easy.
The sharper half of the essay is about what happens next. If a goal can be defined, scored and measured, Krishnan expects machines to surpass people at it quickly. He applies this to AI research itself, arguing that building models can be split into smaller jobs such as hunting for failures, gathering data, running experiments, tuning infrastructure, and serving the result. Each job can be scored, so each can be improved by machines. Improving at improving, he writes, is itself something that can be improved.
That leads to his central claim. The thing machines cannot hand back to us is the choice of target. A system trained to make money in markets will develop a different kind of capability from one trained to cure disease or write software on request. Even inside one field, a slightly wrong goal can yield a system that is very capable and not at all what anyone wanted. The test-gaming agent is his example of how a shortcut nobody anticipated becomes the thing the system learns.
He adds a point that keeps the argument from being only about machines. People are shaped by what gets measured, too. Students study the subjects that standardised exams reward, and he notes that in eras when Latin signalled status, students spent years on Latin. Labor markets pull ambitious people toward some problems and away from others. Measuring a skill, in his view, shapes who develops it.
Krishnan also writes about his own corner of this. He refers to Vals AI’s work on benchmarks, which he describes as two parts: the problems a model is shown, and the rule that judges each response right or wrong. Both choices, he says, carry a worldview. A narrow or contrived set of problems produces systems that ace the exam and break outside it. A reward that captures the wrong thing produces a system that gets better in exactly the wrong direction. The post states no title or role for him beyond that reference, and it does not say how Vals AI’s own tests handle this.
His closing line is the one worth keeping. The most important job left to people, he suggests, may be choosing the slopes machines should be sent up and designing them well, not racing the machines on every slope. Human intelligence has mostly levelled off, he says, while models still have taller summits ahead.
It helps to be clear about what kind of document this is. The essay is a conceptual argument. It offers no study, no measurement and no data on how often badly chosen targets derail real systems, and the claim that choosing objectives will be the last durable human job is a forecast, not a finding. Some of its pieces are also contestable. Go has a crisp winner, while most valuable work does not, and Krishnan himself concedes that some valuable skills resist being pinned down as a clean, scoreable task.
Still, the practical edge is concrete for anyone who buys or builds AI systems. Before the next vendor demo, ask what exactly the system was scored on, who chose that score, and what a clever shortcut against it would look like.
Rayan Krishnan, essay posted on X, 2 October 2026.
