Google DeepMind has published the technical report behind Gemma 4, and it answers a question AI Insiders’ earlier coverage of the quantized, phone-ready weights left open: how the company got a 12 billion parameter model to run without dedicated vision or audio encoders. The paper, posted to arXiv under the Gemma Team byline, lays out a five-model suite ranging from an effective 2.3 billion parameters up to 31 billion, plus a 26 billion parameter Mixture-of-Experts variant that activates roughly 4 billion parameters per token. For teams running open weights instead of renting a frontier API, the report matters more than the release blog post, because it specifies the exact trade-offs behind the speed and memory numbers Google is advertising.
The most unusual choice sits in the 12B model. Every other Gemma 4 size keeps a separate, frozen vision encoder (150 million or 550 million parameters) and a separate 305 million parameter audio encoder built on Google’s Universal Speech Model. The 12B model discards both. Raw image patches get projected into the language model’s embedding space through a single 35 million parameter matrix rather than a full vision transformer, and raw audio chunks are fed in directly with no encoder in between. Google DeepMind frames this as cutting memory fragmentation, not just parameter count: one shared embedding space is simpler to route through than three separate encoder outputs.
The 26B-A4B model is Gemma’s first production Mixture-of-Experts release, activating a fraction of its total parameters per forward pass to hold down inference cost. Both that model and the 31B dense flagship ship with a thinking mode, a reasoning trace generated before the final answer, similar in concept to OpenAI’s o1 approach. The effect shows up concentrated in math and code: Gemma 4 31B scores 89.2 on AIME 2026 without tool use, against 20.8 for the non-thinking Gemma 3 27B, according to Google DeepMind’s own benchmark table.
Those numbers, like most of the report’s benchmark comparisons, come from Google’s internal evals against its own prior generation. The company also reports a Chatbot Arena Elo of 1451 for Gemma 4 31B, placing it 43rd on that leaderboard and, by Google’s account, the highest-scoring dense open-weight model there. Arena’s ranking comes from blind human voting rather than a lab-run eval, which makes it the more credible figure in the paper. The head-to-head benchmark tables against Gemma 3 are still the company grading its own homework.
The long-context section is where the report gets specific about engineering rather than marketing. Gemma 4 holds a 5-to-1 ratio of local sliding-window attention layers to full global attention layers (4-to-1 in the smallest model), reuses attention keys as values inside the global layers, and applies a rotary position encoding variant tuned for long sequences. Combined with cache sharing across layers, Google DeepMind says these choices cut the memory set aside for global attention by as much as 37.5 percent, the single change with the most direct effect on how much context a given chip or phone can hold. The full suite ships under an Apache 2.0 license, which permits commercial fine-tuning and redistribution without the usage caps attached to Meta’s Llama family or Mistral’s non-commercial release tiers.
For teams already running open-weight models in production, the report’s real news is architectural rather than promotional: a 12B model with no dedicated encoders that Google claims beats Gemma 3 27B on vision benchmarks with under half the parameters is worth putting through an internal eval against whatever vision-language stack is currently deployed. Treat the Arena Elo ranking as the trustworthy signal here and the internal benchmark table as a starting point, not a substitute, for that comparison.
Based on Google DeepMind’s Gemma 4 technical report on arXiv.
