NVIDIA open-sources an 11B voice model with live tool calling

NVIDIA published open weights for NemotronLabs VoiceChat, an 11 billion parameter model built to listen, talk and call external tools without switching between separate systems. It is the third full-duplex voice system AI Insiders has covered in a week, after Microsoft’s unannounced MAI Realtime and OpenAI’s shipped GPT-Live, and the first of the three to arrive as downloadable weights rather than an API. The model pairs a Fast Conformer speech encoder with a Nemotron Nano V2 9B language model backbone and an NVIDIA text-to-speech decoder, plus a separate channel dedicated to predicting tool-call scripts.

Cascaded voice stacks route audio through three separate models: a speech recognizer, a text language model, and a text-to-speech engine, each one handing its output to the next. NVIDIA frames NemotronLabs VoiceChat as a replacement for that hand-off chain, running recognition, reasoning and generation inside one set of weights. Collapsing the pipeline matters for two reasons. It removes the sequential delay of three model calls run back to back, and it keeps the reasoning model working from raw audio rather than a flattened transcript, the step where a cascaded system throws away tone, pacing and the timing cues a listener relies on to know when to jump in.

NVIDIA measured turn-taking latency at roughly 450 milliseconds on its Full-Duplex-Bench 1.0 evaluation, rising to 480 milliseconds when the model has to stop mid-sentence because a user interrupted it. On VoiceBench, the standard test for speech-based assistants, it ranks second among open full-duplex models. On Full-Duplex-Bench 1.0 it also ranks second among open models overall. The card does not include a head-to-head score against GPT-Live or MAI Realtime, so how NemotronLabs VoiceChat compares to either closed rival on the same conversation is not something NVIDIA has published.

The more consequential number sits in the tool-calling benchmarks. Among open full-duplex models, NVIDIA says this is the first that can call an external function without leaving the conversation, using a dedicated output channel that predicts a tool-call script while the main channel keeps generating speech. On AU Harness, a spoken adaptation of the BFCL-v3 tool-calling suite, it averages 56.1 percent across simple, multiple, parallel and irrelevant-call scenarios. On Full-Duplex-Bench v3 it picks the right tool 82.5 percent of the time, gets the arguments right only 44.2 percent of the time, and completes an entire call correctly on the first try just 33 percent of the time.

Those numbers mark the gap between a voice demo that answers trivia and a voice agent that can safely check an order status or book a meeting. A model that has to fall silent, hand the request to a text model, and resume once the function returns cannot hold a natural exchange while the tool runs. NemotronLabs VoiceChat instead speaks a short holding line the instant it triggers a call, according to the model card, then continues the conversation once the result comes back. That detail is what separates tool calling as a shippable feature from tool calling as a demo.

The license is where NVIDIA actually differs from the closed alternatives, and it is worth reading closely. NemotronLabs VoiceChat runs under NVIDIA’s OpenMDW terms, version 1.1, and the model card states outright that the model is meant for research use, not production deployment. That is a narrower grant than a permissive license such as Apache 2.0. Anyone can download and inspect the weights, but the license as written does not clear the model for the kind of commercial product Microsoft and OpenAI are building toward with their own closed systems.

Running the model also takes real infrastructure. NVIDIA lists support for A100, H100, H200, B100, B200 and RTX 6000 GPUs, Linux as the only supported operating system, and vLLM as the serving engine, with H100 as the hardware NVIDIA used for its own testing. An 11 billion parameter model that processes audio on both ends is heavier to serve than a text model of the same size, and every figure in the model card came from data center silicon, not the GPU on a typical developer’s desk.

What the model card cannot answer is how NemotronLabs VoiceChat behaves once real traffic, not a curated benchmark, is running through it. NVIDIA gives no latency figure under concurrent load and no independent comparison against GPT-Live or MAI Realtime on identical prompts. Whether 11 billion parameters of speech model is practical to run outside a data center depends on a number the card does not supply: how many simultaneous conversations that GPU list can actually serve.

Teams evaluating full-duplex voice should treat NemotronLabs VoiceChat as the option worth testing for on-premises or fine-tuned builds, not yet as a drop-in substitute for GPT-Live or MAI Realtime, until NVIDIA or an independent benchmark publishes a load-tested latency number and a commercial license.

NVIDIA published the NemotronLabs VoiceChat model card on Hugging Face, listing a release date of August 3, 2026.