Netflix’s AI Platform team published a detailed account of the LLM serving system it operates inside its own production environment, rather than routing traffic to OpenAI, Anthropic, or another hosted provider. The company runs vLLM on top of NVIDIA’s Triton Inference Server, wired into the same Java control plane and gRPC pipeline that already scores its recommendation and ranking models. The decision to self-host at this scale, instead of paying per token to a frontier lab, is a data point on where large engineering organizations now draw the build-versus-buy line for AI infrastructure.
The engine choice changed mid-flight. Netflix’s platform launched on TensorRT-LLM, picked because it was already wired into Triton inside Model Scoring Service, the shared backend that also serves XGBoost, TensorFlow, and PyTorch models. By summer 2025 the open-source field had closed much of the performance gap with specialized engines, and Netflix’s workload by then also spanned embedding generation plus custom per-step constraint logic. A re-benchmark against that mix pushed the team to vLLM, citing quicker iteration on nonstandard model architectures, hooks for extending the decoding logic, and simpler debugging than a compiled engine offered.
Packaging carried a real trade-off inside Triton. One option freezes input and output shapes into the model artifact at build time, meaning any frontend change that touches those shapes forces a coordinated update to the packaging code. The alternative, Triton’s vLLM backend, ships only a small config file pointing at model weights and a tokenizer, with shapes generated automatically each time the model deploys. That flexibility has a cost: when a newer Triton release imported a vLLM module that a later vLLM version had removed, the backend refused to start at all, forcing Netflix to pin compatible engine and server versions inside its build image.
On the API side, Netflix standardized on an OpenAI-compatible HTTP interface as a second front door alongside its internal gRPC path, reasoning that most inference tooling, evaluation frameworks, and client libraries across the industry already speak that schema. Reusing NVIDIA’s OpenAI-compatible Triton frontend surfaced a silent defect: a field controlling structured output was accepted at the API layer but never forwarded to vLLM, so a caller asking for JSON could get malformed text back with nothing flagging the failure. Netflix forked and patched the frontend so that field now maps into vLLM’s own guided-decoding controls.
Rollouts split into two paths. One shifts live traffic between an old and new model version in stages and reverts automatically if a stage fails, but it breaks when a new version changes input or output shapes, because callers keep sending old-shape requests during the cutover window. The other keeps every model version running side by side until each consuming team is ready to switch, at the cost of paying for duplicate GPU capacity during the overlap.
The clearest illustration of what “production revealed” looks like sits inside constrained decoding. Netflix’s first implementation ran each request’s output constraints on CPU, one request at a time, serialized by Python’s global interpreter lock, so enforcement cost rose in step with batch size even though the GPU forward pass stayed fast. A move to vLLM’s newer engine version in the fourth quarter of 2025 pushed that logic to the batch level and into multithreaded C++, flattening the cost curve, but it also introduced new edge cases around partially processed prompts and evicted requests that the earlier design never had to handle.
For engineering leaders weighing hosted APIs against in-house serving, Netflix’s writeup prices out what self-hosting actually costs in practice: strict version pinning, dual deployment paths, and patches to open-source frontends most teams treat as black boxes. Any team sizing an inference build should budget for that operational tax, not just the GPU bill, before assuming self-hosting beats a per-token API at their volume.
Netflix detailed its in-house LLM serving architecture in a Netflix Technology Blog post published July 17, 2026.
