MuonGS Optimizer Delivers 15 Percent Speedup for LLM Pretraining

The Science Behind Smarter Optimizers

Training large language models is one of the most computationally expensive endeavors in modern artificial intelligence. Every percentage point of efficiency gained in the training process translates to millions of dollars saved and weeks of GPU time recovered. A new optimizer called MuonGS, published on October 27, 2026, by researchers Athanasios Glentis, Dawei Li, and Mingyi Hong, promises a 15 percent token-equivalent speedup over the previous state-of-the-art MuonH optimizer — and it does so by rethinking something deceptively simple: how weight norms are controlled during training.

Why Optimizers Matter More Than You Think

When most people think about machine learning breakthroughs, they picture bigger models, more data, or novel architectures. But the optimizer — the algorithm that decides how to adjust model weights based on training signals — is arguably just as critical. Adam, the dominant optimizer since 2015, works by maintaining per-parameter adaptive learning rates. It is reliable and well-understood, but as models have grown into the hundreds of billions and even trillions of parameters, researchers have questioned whether Adam is still the best tool for the job.

The Muon family of optimizers emerged as a compelling alternative. Instead of treating each parameter independently, Muon operates on entire weight matrices using a mathematical technique called Newton-Schulz orthogonalization. This matrix-level approach captures structural relationships between parameters that coordinate-wise methods like Adam miss entirely. The result has been faster convergence and better final model quality in several pretraining benchmarks.

The Weight-Norm Problem

But Muon has a persistent weakness. Without explicit weight-norm control, the model’s weight matrices tend to grow unboundedly during training. This growth reduces the effective learning rate — the ratio between the nominal learning rate and the weight norm — slowing progress as training advances. Researchers addressed this with two approaches: MuonW, which applies weight decay, and MuonH, which hard-clamps the weight norm to a fixed value. MuonH became the practical favorite, but both solutions share a limitation: they restrict the model’s ability to explore the loss landscape by constraining how far weights can move from their initialization.

Enter MuonGS: A New Approach to Norm Control

The key insight behind MuonGS is that you do not need to fix the weight norm to control it. Instead of clamping or decaying, MuonGS uses Gram-Schmidt projection to remove the radial component of each Muon update — the part that changes the weight matrix’s overall scale — while preserving the tangential component that changes its direction. This allows the weight norm to grow, but at a controlled rate proportional to the square root of the number of training steps.

This might sound like a minor mathematical adjustment, but it has profound implications for how the optimizer accumulates effective learning rate over the course of training. The researchers formalized this through a quantity they call the accumulated effective learning rate, defined as the sum of the effective learning rate across all training steps. This metric captures how much cumulative adjustment the optimizer can make to the model relative to its current scale.

  • Vanilla Muon accumulates only logarithmically, meaning progress slows dramatically over time
  • MuonW and MuonH guarantee at least linear accumulation by constraining the norm
  • MuonGS achieves square-root accumulation — slower than linear but far faster than logarithmic — without fixing the norm

The square-root growth rate is the sweet spot. It is fast enough to sustain meaningful exploration throughout long training runs, yet slow enough to maintain stability. The researchers proved mathematically that for smooth scale-invariant objectives, MuonGS achieves an O(T^{-1/2}) stationarity guarantee — meaning it reliably converges toward optimal solutions.

Experimental Results: Where Theory Meets Practice

Theoretical guarantees are necessary but not sufficient. The real test is whether MuonGS actually performs better when training real language models. The researchers ran pretraining experiments across three model configurations:

  • 160M-parameter dense model — a compact architecture suitable for rapid experimentation
  • 400M-parameter dense model — a mid-scale model that reveals scaling behavior
  • 135M active / 1B total parameter MoE model — a mixture-of-experts architecture that reflects how production models are increasingly built
  • What the Numbers Show

    Across all three configurations, MuonGS achieved lower validation perplexity than MuonH at long training horizons. Perplexity measures how well a language model predicts the next token — lower is better. For the dense models, the improvement translated to an estimated token-equivalent speedup of approximately 15 percent at matched validation loss. This means that to reach the same model quality as MuonH, MuonGS needs roughly 15 percent fewer training tokens — a massive saving when training runs consume trillions of tokens.

    The MoE results are particularly significant. Mixture-of-experts architectures, which activate only a subset of parameters per token, have become the backbone of frontier models like DeepSeek-V4-Pro and Kimi K3. Demonstrating that MuonGS works on MoE architectures confirms that the optimizer is not limited to academic-scale dense models but is relevant to the architectures powering the most advanced AI systems in production.

    The Broader Context: A Wave of Optimizer Innovation

    MuonGS arrives amid a surge of optimizer research. In 2026 alone, multiple papers have proposed improvements to the Muon framework. MeqMuon introduced matrix-equilibrating techniques to balance update magnitudes across weight matrix rows. SOAP explored preconditioned gradient methods at larger pretraining scales. NVIDIA researchers validated that Muon’s large-batch efficiency transfers to MoE architectures at the 5×10^22 FLOP scale.

    This collective effort reflects a growing recognition that optimizer design is an underexplored frontier. For years, the machine learning community treated Adam as a solved problem and focused architectural innovation. But as pretraining costs balloon — DeepSeek-V4-Pro required 33 trillion tokens, and Kimi K3 reaches 2.8 trillion total parameters — even single-digit percentage improvements in training efficiency translate to enormous cost savings.

    What Comes Next

    Several open questions remain. The MuonGS experiments were conducted at relatively modest scales — up to 400M dense parameters. Whether the 15 percent speedup holds at the billion and trillion-parameter scales used by frontier labs remains to be verified. The researchers also note that their analysis assumes smooth, scale-invariant objectives, which is an approximation of the complex loss landscapes encountered in practice.

    Nevertheless, MuonGS represents a meaningful step forward in optimizer theory and practice. By identifying the accumulated effective learning rate as the key metric and showing that relaxed norm control can outperform strict constraints, the work opens a new design space for future optimizers. As the AI industry grapples with the escalating costs of training ever-larger models, innovations like MuonGS at the optimizer level may prove just as impactful as architectural breakthroughs at the model level.

    Key Takeaways for ML Practitioners

    • Optimizer choice matters — switching from Adam to Muon-family optimizers can yield measurable pretraining speedups
    • Norm control is a spectrum — neither unconstrained growth nor hard clamping is optimal; controlled growth via Gram-Schmidt projection offers a better tradeoff
    • Accumulated effective learning rate is a useful diagnostic for understanding optimizer behavior over long training horizons
    • MoE compatibility — MuonGS works on mixture-of-experts architectures, making it relevant for modern production model designs
    • Open research direction — scaling MuonGS to frontier model sizes and exploring alternatives to Gram-Schmidt projection are promising next steps

    As machine learning continues to scale, the humble optimizer — long overshadowed by attention mechanisms, mixture-of-experts routing, and multimodal architectures — is finally getting the attention it deserves. MuonGS proves that there is still substantial room for improvement in how we train the models that increasingly shape our digital world.


    Edited by Palawan @QUE.COM
    Website: https://QUE.COM Intelligence
    Sponsored by: https://MAJ.COM AI Autonomous


    Discover more from QUE.com

    Subscribe to get the latest posts sent to your email.

    Leave a Reply

    Connect with

    Discover more from QUE.com

    Subscribe now to keep reading and get access to the full archive.

    Continue reading