An engineer who posts as ali, known as @waterloo_intern, published a long technical write-up on 27 July titled “From GPT2 to Kimi3, Explained.” Ali works on inference at Baseten and studies engineering at the University of Waterloo. The piece traces the architectural lineage connecting OpenAI’s original GPT-2 to Kimi K3, the latest release from Moonshot AI’s Kimi line.
The central argument is that Kimi K3’s improvements do not come from scale alone. According to the write-up, the model combines four mechanisms: constant-state Kimi Delta Attention, periodic softmax retrieval, sparse experts, and selective residual access. Each step in the lineage, ali argues, changes how a fixed-capacity memory stores, forgets, and retrieves information while keeping inference efficient.
The underlying concepts are established ones. Standard softmax attention compares every token against every prior token, which gets accurate but expensive as context grows. Constant-state, or linear, attention compresses that history into a fixed-size state that updates as new tokens arrive, trading some precision for a memory footprint that does not balloon with sequence length. Periodic softmax retrieval reintroduces full attention at intervals, a way to recover detail the compressed state would otherwise lose. Sparse mixture-of-experts routes each token to a subset of specialized parameters instead of the full network, cutting compute per token. Selective residual access controls which layers a signal can skip forward to, shaping what earlier computation survives into later stages.
What ali’s write-up frames as the throughline is that each generation since GPT-2 has been less about adding parameters and more about redesigning how the model manages a limited memory budget under inference constraints. That framing cuts against the industry’s default explanation for capability gains, which leans on raw scale: more parameters, more data, more compute. If the lineage ali traces holds, architecture choices about storage, forgetting, and retrieval are doing comparable work to parameter count, and doing it at lower serving cost.
The post spread quickly, passing 2.4 million views, with readers describing it as one of the best explanations of a transformer model’s lineage they had encountered. That reception says something about the appetite among practitioners for architectural detail over marketing claims about benchmark scores.
For builders evaluating inference costs on long-context workloads, the practical takeaway is to treat attention mechanism choice as a first-order decision, not an implementation detail. Teams comparing Kimi K3 against other frontier options over the next quarter should ask vendors directly how their models handle the tradeoff between constant-state compression and periodic full-attention recovery, since that tradeoff determines both cost per token and how much long-range detail survives a long conversation.
Based on a technical write-up titled “From GPT2 to Kimi3, Explained,” published on X by ali (@waterloo_intern) on 27 July 2026.
