Most multimodal models borrow their eyes from a vision encoder trained separately on images, and a new study argues that the borrowing is the least important part. The project page, “Unbundling Multimodal Vision Encoding,” comes from a twelve-author team including Ellis Brown, Chris Hoang, Luke Zettlemoyer and Saining Xie. It finds that what counts is whether vision gets its own dedicated parameters, not where those parameters came from.

The authors split an encoder into three bundled choices: a visual prior from separate pretraining, dedicated vision capacity, and placement of that capacity before the language model fuses image and text. They trained every model from scratch on the same data, backbone and recipe, then compared recipes at matched lifetime compute. That accounting charges a recipe for every stage, including encoder pretraining. SigLIP 2, for example, trains on 40 billion image-text samples, a cost most comparisons leave out.

The first result is that the pretrained prior is optional. An encoder trained alongside the language model matched encoders pretrained four different ways. At the largest 4B budget, a MetaCLIP-pretrained encoder trailed the scratch encoder by 1.8 points on average visual question answering (VQA). More pretraining raised ImageNet accuracy but not VQA, even when the pretraining was free.

The second result cuts the other way. Encoder-free models, which pass raw image patches directly to the language model, lagged on visual understanding. At 1.7B parameters they trailed by 4.1 to 6.4 VQA points, and across backbones from 1.7B to 6.4B they needed about three times the compute to catch up. Handing them the encoder’s 0.4B extra parameters as width or depth moved VQA by at most 0.4 points. A bigger model was not the fix. Reserved visual capacity was.

The third result is the one builders may care about most. The dedicated capacity does not need to sit in front of the language model. Vision-only modules in the first eight of 28 backbone layers, which the authors call in-fusion, scored 45.0 on average VQA against 44.4 for the scratch encoder. The same budget placed in other positions did no better than encoder-free. Early placement helped most on OCR and chart questions, where fine detail decides the answer: 36.0 versus 25.2 for encoder-free.

Serving cost is where the design pays off. A standalone encoder has to process an image first, and the backbone waits, so latency stacks. Measured on a single H200 with 16 simultaneous requests, time to first token was 37 milliseconds for in-fusion at 0.4B vision-only parameters, against 64 for the encoder. Copying every backbone layer for vision tokens cost 100 milliseconds and scored lower.

The findings held when the team grafted vision onto an existing text-only model of 1.7B parameters. In-fusion scored 50.1 on average VQA, the scratch encoder 48.6, and encoder-free 39.8.

The page carries limits worth reading. The team’s encoders match common released ones in size but see far less data, so the results leave open whether the larger data diet of released encoders like SigLIP 2 buys an advantage. The experiments top out at backbones of a few billion parameters, and the page offers no test at frontier scale. A study built on the authors’ own training setup is also not an independent replication.

For teams planning a multimodal model with a fixed compute budget, the study suggests spending less on pretraining a standalone encoder and more on vision-only layers inside the backbone, then checking whether that design also trims latency on their own serving hardware.

Source: “Unbundling Multimodal Vision Encoding,” a research project page published at unbundling.github.io by Ellis Brown, Chris Hoang and co-authors, November 2026.