NVIDIA’s Sigma Drafts Text in Parallel Blocks and Runs About 3.5 Times Faster Than Standard Decoding, Making Diffusion a Real Contender for Cheaper AI Inference

For most of the generative AI era, large language models have written the way a person types: one token at a time, each word waiting on the one before it. That sequential habit is a big reason inference is slow and expensive at scale. A research team from NVIDIA, Cornell University and Georgia Tech is making a serious case for a different approach. In an updated version of their paper posted to arXiv on November 12, the group details Nemotron-Labs-Sigma, a family of 3-billion and 8-billion-parameter “continuous diffusion” language models that draft whole blocks of text in parallel and, in the authors’ tests, deliver roughly 3.5 times the throughput of standard autoregressive decoding while keeping that model’s answers.

The researchers describe Sigma as the first large-scale continuous diffusion language model, and the paper says the model weights are publicly available. For businesses watching their AI bills, the more important takeaway is the direction: the race to make models cheaper to run is no longer only about smaller models and better chips. It is also about changing how models generate text in the first place.

How Sigma Writes Differently

Diffusion models became famous in image generation, where a system starts with random noise and refines it, step by step, into a picture. Applying the same idea to language is harder because words are discrete symbols, not smooth pixel values. Most text diffusion models to date have worked by “masking” tokens and filling them back in, which the authors argue creates a jagged, high-dimensional space that is hard to steer.

Sigma takes a different route. According to the paper, each token is represented as a point in a compact 16-dimensional embedding space. The model adds Gaussian noise to a block of these embeddings and learns to clean it up, guided by the text that came before. When a block is finished, it is converted back into words and added to the context, and the next block begins. To keep training costs manageable, the team warm-started Sigma from existing pretrained autoregressive model weights rather than training from scratch.

Because the path from noise to text is smooth, the researchers say it can be steered at inference time. Two controls, classifier-free guidance and a setting they call score temperature, let operators trade off answer quality against diversity. A summary of the paper on alphaXiv notes the authors packed both guidance branches into a single forward pass, reporting less than 5% added wall-clock time per denoising call at block sizes of 32 and 64 tokens.

The Numbers Behind the Speed Claims

The headline efficiency results come from two techniques. The first is distillation, in which a slower “teacher” model trains a “student” to reach similar answers in far fewer refinement steps. The authors report roughly 8x acceleration on math and 4x on coding, equivalent to producing 8 and 4 tokens per forward pass, with what they describe as negligible accuracy loss and preserved diversity.

The detailed figures show where that holds and where it slips. Per the alphaXiv summary, the distilled 3B model running just 8 denoising steps scored 84.62% on the GSM8K math benchmark, against 85.57% for its full teacher run at 256 steps. On HumanEval coding it scored 56.40% versus 57.26%. But on the MBPP coding benchmark, the gap widened to 49.04% versus 58.00%, a real cost the authors acknowledge.

The second technique is self-speculative decoding. Sigma’s diffusion side drafts a 32-token block in one pass, and the same model’s autoregressive side checks the draft, accepting the longest stretch that matches what it would have written itself. Because the final output is that model’s own greedy autoregressive answer, the speedup does not change the result. Measured on a single NVIDIA GB200 using SGLang with one request at a time, the 8B model averaged 3.46x throughput over its own autoregressive decoding across GSM8K, HumanEval and MBPP, compared with 3.04x for NVIDIA’s earlier masked-diffusion model, Nemotron-Labs-Diffusion. Per-task speedups were 4.20x, 2.72x and 3.47x respectively.

Accuracy: A Modest Edge, Not a Knockout

On quality, Sigma is competitive rather than dominant. After pretraining, the regularized Sigma-8B averaged 71.16% across six math and coding benchmarks, versus 70.22% for the matched-size Nemotron-Labs-Diffusion-8B, while trailing it on MBPP and MBPP+, according to the alphaXiv summary. Where Sigma does stand out is when many answers are sampled: a NeuralTrendHub review highlights an MBPP pass@256 score of 91.8%, compared with 87.6% for Nemotron-Labs-Diffusion, a sign the model can find more distinct correct solutions when given more tries.

There are important caveats. These are 3B and 8B research models, not frontier-scale systems. The throughput figures were measured under a single-request serving setup, which is not the same as a busy production cluster handling many users at once. And reinforcement-learning post-training, a standard step in making today’s leading models useful, is left as future work.

Why It Matters for Business

Inference, not training, is where most companies feel AI costs every month. Every chatbot reply, code suggestion and agent step is billed by the token or by the GPU-hour. A technique that can produce several tokens per forward pass without changing the answer goes straight at that cost line. If speedups of this kind survive at larger scales and under real traffic, the same hardware could serve more customers, or the same workload could run on less hardware.

Speed also changes what is practical. Real-time applications such as voice assistants, live customer support and coding tools are especially sensitive to latency, because users notice every pause. Faster generation of math and code, the two areas where Sigma’s gains are reported, maps directly onto agentic workflows that chain many model calls together.

Strategically, the work fits a pattern for NVIDIA: publishing open research and weights that make its GPUs more productive keeps developers building on its stack. For buyers, the practical move today is not to switch architectures, but to track whether diffusion-style decoding lands in mainstream serving engines and model APIs, and to ask vendors how they plan to pass inference savings along.

The Bottom Line

Nemotron-Labs-Sigma does not dethrone the autoregressive model, and its strongest speed results come with clear testing limits. But it shows that continuous diffusion can match a strong baseline on reasoning and coding while opening new routes to faster, steerable generation. In a market where the cost of every token matters, that makes it a research direction worth watching closely. Additional analysis is available from Saluca.

Edited by Coron @QUE.COM
Website: https://QUE.COM Intelligence
Sponsored by: https://MAJ.COM AI Autonomous