Why Default Optimizer Settings Are Sabotaging Your Machine Learning Models

Machine learning practitioners have a dirty little secret. Most of us never touch our optimizer settings. We import Adam or AdamW, accept the default parameters without question, set the learning rate to 3e-4, and hope for the best. This approach, popularized by years of community convention and reinforced by AI code assistants that generate the same boilerplate, works often enough to lull us into complacency. But when it fails, it fails spectacularly, and the culprit is rarely where you think to look.

The Adam Optimization Illusion

Adam, introduced by Kingma and Ba in 2015, became the dominant optimizer in deep learning for good reason. It combines two powerful ideas: momentum, which smooths noisy gradients by maintaining an exponentially weighted moving average of past updates, and adaptive learning rates, which scale each parameter’s step size based on the historical magnitude of its gradients. The result is an optimizer that converges quickly, tolerates noisy data, and largely frees practitioners from the painstaking learning rate tuning that vanilla stochastic gradient descent demands.

The problem is that this reliability has bred a dangerous complacency. Engineers treat optimization as a solved problem. They focus on architecture, data pipelines, and loss functions while the optimizer sits in the background, quietly running with settings that may be catastrophically wrong for their specific use case. Andrej Karpathy’s famous 2016 quip that 3e-4 is the best learning rate for Adam became dogma. Nearly a decade later, practitioners still recite it like scripture.

How Adam Actually Works

To understand where Adam goes wrong, you need to understand what it actually does under the hood. Adam maintains two running statistics for every parameter in your model. The first moment estimate, controlled by the beta-one parameter, tracks the exponentially weighted average of gradients. This is the momentum that gives the optimizer its directional persistence. The second moment estimate, controlled by beta-two, tracks the exponentially weighted average of squared gradients. This is the variance estimate that determines how each parameter’s update is normalized.

The default beta-one of 0.9 means the momentum looks back roughly over the last 10 steps. The default beta-two of 0.999 means the variance estimate averages over approximately the last 1,000 steps. This asymmetry is deliberate. Direction changes more frequently than magnitude, so it makes sense to let momentum react faster than variance. But this 1,000-step memory window is the source of many of Adam’s most spectacular failures.

Where Adam Fails Spectacularly

The failures are not theoretical edge cases. They are documented, reproducible, and affect some of the most common scenarios in modern machine learning.

Convergence Failures on Simple Problems

In 2018, Reddi, Kale, and Kumar published a landmark paper at ICLR demonstrating that Adam fails to converge on certain simple convex optimization problems. The original Adam paper contained a convergence proof that was, quite simply, wrong. The authors constructed an explicit example where Adam’s short-term memory causes informative gradients to be discarded too quickly, leading the optimizer to converge to the worst possible solution. This is not a subtle failure. The optimizer confidently marches in the wrong direction.

Worse Generalization Than SGD

Wilson et al.’s 2017 NeurIPS paper showed something even more troubling. On simple classification tasks, vanilla SGD achieved zero test error while adaptive methods including Adam produced significantly larger errors. The adaptive scaling that makes Adam so convenient during training can actually hurt generalization. The mechanism is straightforward: by giving frequently-updated parameters smaller effective learning rates and rarely-updated parameters larger ones, Adam can amplify the importance of spurious features that happen to receive small gradients early in training.

Loss Spikes in Large Language Model Training

Perhaps the most relevant failure mode for today’s practitioners comes from Meta’s research on large-scale training. Molybog and colleagues documented unexplained loss spikes during training of language models ranging from 7 billion to 546 billion parameters. Their analysis traced the problem directly to Adam’s adaptive gradient rescaling. When some parameters enter a state where gradient updates are consistently small, the second moment estimate shrinks over hundreds of steps. If a large gradient suddenly arrives for one of those parameters, dividing by the now-tiny variance estimate produces a gigantic update. The result is a loss spike that seems to come from nowhere, with no obvious bug or data issue to blame.

At scale, this is particularly insidious. Nobody inspects the second-order moment statistics of optimizer states across hundreds of millions of parameters. The spikes appear random, and teams waste days investigating data pipelines, gradient clipping thresholds, and architecture choices when the real problem is hiding in three numbers they never changed.

Reinforcement Learning: Adam’s Toughest Test

If supervised learning is Adam’s comfort zone, reinforcement learning is its battlefield. The core assumption that makes Adam’s adaptive learning rates sensible is that the gradient distribution is roughly stationary. The volatility of a parameter update now should inform the volatility of future updates. In supervised learning, this holds well enough. The data distribution is fixed, the loss landscape is stable, and gradient statistics evolve predictably.

Reinforcement learning shatters this assumption. The policy generating training data 10 steps ago may be completely different from the policy generating it now. The optimization landscape is constantly shifting. The second moment estimate, averaging over 1,000 steps, is always chasing a moving target that has already moved on. In actor-critic methods, the target itself changes as the value network updates, meaning the true gradient direction for any parameter can flip sign within 100 steps while the variance estimate is still averaging over the past millennium.

The Epsilon Hack Nobody Talks About

Deep RL researchers have quietly worked around Adam’s pathologies for years, but the solutions live in source code rather than publications. Google’s Dopamine framework, a reference implementation for deep RL, uses an epsilon value of 1.5e-4 for DQN and Rainbow agents. Adam’s default epsilon is 1e-8. That is a difference of roughly 15,000 times. For IQN and M-IQN agents, the epsilon is set to 3.125e-4, over 31,000 times larger than the default.

This epsilon value is the stability floor that prevents division by near-zero when the variance estimate shrinks. In RL, where gradient magnitudes can collapse v_t to vanishingly small values, a larger epsilon is not a minor tweak. It is the difference between training and not training. Yet this critical adjustment is almost never discussed in research papers. You have to read the framework source code to discover it.

Practical Guidance: When and How to Break the Defaults

The answer is not to abandon Adam. Research from Obando-Ceron and Castro in 2021 directly compared Adam against RMSProp across 60 Atari games and found that Adam with MSE loss actually outperformed the standard RMSProp plus Huber loss configuration that dominated RL labs from 2015 to 2018. Adam can work in challenging environments. It simply needs to be coaxed.

Here is a diagnostic framework for when to deviate from the defaults:

• Large unexplained loss spikes: Reduce beta-two from 0.999 to 0.99, then to 0.95 or lower. This shortens the second moment memory window and lets the variance estimate react more quickly to changing conditions.
• Highly non-stationary problems, especially RL: Increase epsilon substantially (try 1e-4 or higher) and reduce beta-two. The optimizer needs to forget stale statistics faster and resist division by near-zero variance estimates.
• Noisy gradients causing instability: Increase gradient clipping before reducing the learning rate. The instability may not be the learning rate at all but rather Adam’s adaptive normalization amplifying noise.
• Poor generalization despite excellent training loss: Try swapping to SGD with momentum. Adaptive methods can overfit to training dynamics in ways that vanilla SGD avoids, particularly on simpler tasks.
• Training very large models: Monitor for sudden gradient norm explosions. If they appear, investigate optimizer state. The second moment estimate is almost certainly the culprit.

The PyTorch Default Trap

One final gotcha catches even experienced practitioners. In PyTorch, Adam defaults to a weight decay of zero. AdamW, which implements weight decay as a separate decoupled step rather than through L2 regularization, defaults to weight decay of 0.01. If you swap Adam for AdamW without checking the defaults, you may unintentionally introduce weight decay that subtly changes your training dynamics. The assumption that defaults are consistent across related implementations in the same library is, unfortunately, wrong.

Optimization Is Not a Solved Problem

The optimizer is not a brief line in your training script. It is the primary decision handling every weight update in your model, whether that model has a thousand parameters or a hundred billion. Treating it as solved because the defaults work on your first few experiments is a gamble that will eventually cost you.

The next time you encounter inexplicable loss spikes, unstable reinforcement learning training, or a model that refuses to converge despite weeks of architectural experimentation, do not redesign the network first. Open the optimizer configuration. Inspect the hyperparameters. Question the defaults. You might discover, as many researchers have after burning thousands of GPU hours, that the problem was never your architecture at all. It was three numbers you never thought to change.

---

Edited by Palawan @QUE.COM
Website: https://QUE.COM Intelligence
Sponsored by: https://MAJ.COM AI Autonomous