After a driver upgrade, one of Netflix’s health checks for its GPU machines stopped running. For days, servers carrying the wrong count of network adapters entered service, and only the slow jobs they produced led engineers back to the cause. The upgrade had removed a program the adapter check depended on. The check did not fail. It simply went quiet, which looks exactly like a machine that passed.
Netflix’s engineers describe this in a post on the Netflix Technology Blog, written by Nick Rutigliano, Argha C and Samartha Chandrashekar, and their own verdict is the line worth keeping: “A check that can silently stop running is worse than no check, because it gives false confidence while the machine keeps working.” Think of a smoke detector whose battery has died. With no detector, you stay alert. With a dead one on the ceiling, you stop looking. Netflix’s fix has two parts. A missing dependency now makes the check fail, and when a driver or diagnostic tool hangs so the hardware cannot be queried, the machine is marked unhealthy rather than passed.
The reason this matters more for GPUs than for ordinary servers is the shape of the work. A large training job can last several weeks and spans hundreds of GPUs, all moving in lockstep. The job advances at the pace of its slowest participant and stops if one dies. That makes partial failure the expensive kind. Netflix describes a GPU that answers a status query but falters when real work arrives because of a memory or firmware fault, and a network adapter that shows fewer links than it should. With the adapter, the job survives but slowly, on a sliver of the usual bandwidth.
The second admission is a misdiagnosis. A class of timeouts in NCCL, the communication library GPUs rely on, looked like a network fault. The cause was stalls in loading data when many workers ran, and the GPUs and network were healthy. Had Netflix treated those as hardware failures, healthy machines would have been cycled out with nothing fixed. That is why the company sorts every failure into a platform fault or an application fault before it retries anything.
The third is about trusting the vendor. Netflix says the vendor’s catalogs of driver error codes trail the newest GPU generations and sometimes disagree with production behavior, so it maintains its own table of fatal codes for each GPU family. One generation fires off a flood of codes that are normally fatal each time a healthy machine boots. Netflix learned to ignore them instead of draining brand-new machines.
Netflix ties these GPUs to training the models behind its home screen and, in a growing share, serving them live. The post says nothing about video encoding. It also gives no failure rates, no fleet size and no count of how many machines were affected, only that at this scale failures happen often enough to be treated as normal.
The two outcome figures are Netflix’s own and carry no date range or baseline: GPU node availability “consistently” above 99.95%, and job success rising from about 98% to above 99.5%. Since job success here means finishing without a platform-caused failure, that is at least roughly three quarters of such failures gone.
Can you have it? Partly. Repairs run through Karpenter, which Netflix customized with a policy giving each fault its own tolerance window and a cap on how much of a cluster can be disrupted at once. The node agent sits on node-problem-detector, an open-source Kubernetes project, and the device checks draw on NVIDIA’s DCGM. Netflix says it plans to upstream that feature. The checks themselves are not released in the post. The custom scheduler and temperature-gap fault detection it mentions are planned work, not shipped.
Any team running its own GPUs can borrow the cheapest lesson today: break a dependency of one of your health checks in staging and see whether anything complains.
Based on the Netflix Technology Blog post “Reliable GPU compute for large training and inference jobs” of 14 October 2026.
