Sebastian Raschka, the machine learning researcher known for his LLM architecture write-ups, published notes this week reading Kimi K3’s architecture diagram line by line. His conclusion: the design underneath Moonshot’s new open-weight release is not new. It is a production-scale build-out of Kimi Linear, the 48-billion-parameter model the company shipped last year, expanded to 2.8 trillion parameters, a jump of roughly 58 times that makes K3 the largest open-weight model currently available.
That continuity is the more interesting story than the release itself. When a lab’s flagship model is an enlarged version of a design already validated at smaller scale, rather than a fresh architecture, it signals the team believes the underlying approach still has room to grow rather than needing a redesign. Betting a 2.8-trillion-parameter release on a known recipe is a safer wager than debuting new architecture at that size. It also means K3’s ceiling is set by whatever Kimi Linear’s core ideas can actually support once pushed this far.
Raschka flags one genuinely new component: LatentMoE, a mixture-of-experts variant, meaning a model built from many specialized sub-networks that only a subset activate for any given input. LatentMoE shrinks its large linear layers through down-projection, the same compression trick multi-head latent attention (MLA) uses to cut the memory cost of stored attention history. He notes it is the identical LatentMoE design used in Nemotron 3 Ultra, Nvidia’s model, not a Moonshot original.
Most of the remaining changes are efficiency substitutions rather than new capabilities. Standard mixture-of-experts layers become LatentMoE. Regular attention becomes multi-head latent attention. Kimi Delta Attention, a hybrid attention mechanism carried over unchanged from Kimi Linear, handles additional layers. Raschka groups this pattern with recent releases from Nemotron 3 and DeepSeek V4 as evidence of an industry-wide push toward cheaper inference rather than more capable models in the abstract.
One change is not an efficiency tweak: attention residuals, a technique already present in Kimi Linear that links the residual path across layers using an attention score to weight each layer’s contribution. DeepSeek V4 tackled the same residual-path problem differently, widening it with a method called manifold-constrained Hyper-Connections. Raschka reports that, per Moonshot’s technical documentation, attention residuals nudge validation loss and downstream accuracy in a better direction, consistently, at a cost near 4 percent more training compute and 2 percent more inference compute.
The more structurally notable choice is positional encoding, the method a model uses to track word order in a sequence. K3 drops RoPE (rotary position embeddings, the standard technique for this) entirely and runs NoPE (no positional embeddings) throughout the network, a choice inherited wholesale from Kimi Linear. Most recent architectures mix the two, keeping RoPE in local attention layers such as sliding-window attention and reserving NoPE for global layers. Raschka writes that, to his knowledge, K3 is the first frontier-scale model to run NoPE everywhere, a claim he frames as an observation rather than a verified record.
K3 also ships with native multimodal support, a capability Kimi Linear lacked.
For teams evaluating open-weight infrastructure, K3’s architecture reads as a bet that Moonshot has more to gain from scaling a validated recipe than from reinventing it. Whether that bet pays off depends on whether K3’s benchmark gains scale in proportion to its 58-fold size increase, a comparison worth running before committing serving infrastructure to it.
Sebastian Raschka published these architecture notes on his blog on July 28, 2026.
