OpenAI’s Jalapeño inference chip beat Nvidia’s Blackwell on performance per watt in SemiAnalysis InferenceX benchmarks, according to Key Context, Tae Kim’s newsletter. The tests ran Kimi K2.5, DeepSeek R1, and GPT-OSS, all open-source models, and OpenAI presented Jalapeño at Hot Chips in late August. Neither the newsletter nor the interview gives the size of the lead.
Kim adds his own caveats and verdicts, and they are his, not Ho’s. He writes that the rankings could change once Nvidia’s newest Vera Rubin systems arrive. He also calls Jalapeño the strongest program among recent AI chips that are not GPUs, and he calls a high-performing first chip built so quickly a big achievement.
Richard Ho is OpenAI’s VP of Hardware and led the Jalapeño team. He started on Google’s TPU from its first generation, and he later worked at the photonics company Lightmatter before Sam Altman’s people recruited him. Ho says no one else was working on hardware at OpenAI when he arrived, and that Altman first asked for an evaluation, not a chip.
Ho describes the design as a split memory system. The chip has 64 cores, and each one can directly address its own bank of HBM, the stacked high-bandwidth memory next to the processor. Reaching any other bank means going through a network, which is slower. A standard GPU instead shares a single address space across the chip, an arrangement that simplifies programming but costs hardware overhead to keep that memory coherent. Ho says OpenAI dropped that overhead and accepted a harder programming model, betting that AI models would get good at managing the memory map. In his words, the analysis showed the trade-off “paid off big time.”
His explanation for the benchmark results is utilization. Ho says the chip’s peak figures for compute, memory, and networking are comparable to other devices, but “we get this performance boost in real benchmarks because our utilization is very high.” Older designs, he says, ran out of compute during prefill and out of memory bandwidth during decode, leaving parts of the hardware idle.
Two design choices follow from that. Ho says OpenAI chose not to split hardware into separate prefill and decode pools, because a data center operator must commit to that ratio for years, and he prefers one device that works across the whole range. He says it picked HBM over SRAM because SRAM offers bandwidth but not enough capacity for OpenAI’s large models, given what chip manufacturing processes allow. Ho says HBM4 was the right technical choice at the time and that HBM will likely stay in the architecture in some form.
On the schedule, Kim put the move to tapeout (the point where a design is finalized for manufacturing) at nine months in his question. Ho credited AI agents with exploring design optimizations at a scale a human team could not match, and he said the team is about 100 people covering everything from chip design to firmware. He says OpenAI is working on the second and third generations and has started the fourth, but he would not call the cadence annual, since each data center rollout has to deliver a clear step up in performance per watt.
The interview also limits the threat to Nvidia. Ho says OpenAI’s training is almost exclusively on Nvidia GPUs, and that for inference the company will take compute from its own chip, Nvidia, AMD, or others, depending on capacity. For Nvidia, the open question is whether Vera Rubin closes the efficiency gap on inference before OpenAI’s second-generation chip arrives.
Reported by Key Context, Tae Kim’s newsletter, in an interview with OpenAI’s Richard Ho (publication date not stated in the source).
