DeepSeek, the Hangzhou-based lab whose V3 release shipped at a fraction of frontier training costs, has put DeepInfer on GitHub under the Apache 2.0 license. The repository holds the inference software DeepSeek says powers its own large-scale services. It is a partial release: offline inference for one model, DeepSeek-V4.1-Flash, with three pieces of a real serving stack left for users to supply.

What does ship is a dense engine. The repository bundles optimized kernels, a model runtime, and a sampler, built on named components such as FlashMLA, DeepGEMM, and DeepEP, plus custom TileLang kernels. To spread a large mixture-of-experts model across many chips, it supports data-parallel attention and expert parallelism with load balancing between experts.

Several features target decoding speed. DSpark generates draft tokens and uses confidence estimates to decide when to verify them. A bounded-replay mode trims the work the decoder repeats during prefill, keeping only a short suffix of tokens. The scheduler also overlaps CPU batch planning with work on the accelerator, and that mode is on by default.

Two other features matter to anyone who tests models for a living. Batch-invariant inference keeps a model’s results identical no matter what else shares the batch, and deterministic sampling gives the same text on every run when the prompt tokens, the sampling settings, and a fixed seed all stay the same. Both run on NVIDIA hardware only, so the Ascend path is a step behind on reproducibility. Structured outputs, which constrain generation to a regex, a JSON schema, or a grammar, are also supported.

The hardware list is short: NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs. A major Chinese lab publishing a serving stack with a first-class path onto Huawei silicon fits a wider pattern. US export controls limit which Nvidia chips Chinese companies can buy, and a team that can serve its flagship model on domestic accelerators has a hedge against that supply. DeepSeek does not discuss the motive in the repository; the setup instructions simply list Huawei’s CANN software alongside CUDA 13.

The gaps are where the work lives. According to the repository, a complete service still needs a KV cache system (the memory of earlier tokens that lets a model avoid recomputing them), disaggregation of prefill and decode onto separate machines, and a load balancer that spreads traffic over the data-parallel replicas handling each of those two stages. DeepSeek notes that most of the engine functionality for disaggregation is already implemented. The surrounding infrastructure is not.

Those three items are what turn an engine into a product. The repository also reports prefill and decode benchmarks measured on 4 SM100 GPUs and 64 GPUs respectively, each inside a single scale-up domain, and includes a script to reproduce them.

The code has a clear lineage. The acknowledgments section says DeepInfer grew mainly out of vLLM, and borrowed ideas from SGLang and TensorRT-LLM, which means teams already running those engines will recognize much of the design.

For a company weighing whether to self-host V4.1-Flash, the engine is now the cheap part. The budget question is how many engineer-months it takes to build the cache and the routing around it.

Published by DeepSeek on GitHub, in the DeepInfer repository, October 2026.