Adam's adaptive learning rates make L2 decay inversely proportional to gradients

Most engineers treat weight decay and L2 regularization as synonyms. That equivalence collapses under Adam. The optimizer's adaptive scaling reverses the intended effect: parameters receiving large gradients undergo weaker effective decay, while those with tiny gradients get penalized heavily. This is not a flaw. It follows directly from how Adam normalizes each update.

Adam tracks per-parameter first and second moment estimates:

m_t = β1 * m_{t-1} + (1 - β1) * g_t
v_t = β2 * v_{t-1} + (1 - β2) * g_t^2

The update divides the gradient by sqrt(v_t) + ε. Adding an L2 penalty λw to the loss gradient subjects that penalty to the same adaptive denominator. Large g_t produces large v_t, which yields a large denominator, suppressing the λw contribution. Small g_t yields a small denominator, letting the penalty dominate.

This behavior has bitten teams training transformers. Attention heads with strong gradient signals — early layers, high-attention positions — effectively dodge weight decay. Meanwhile, feed-forward neurons with sparse activations get regularized aggressively. The regularization budget distributes unevenly, skewing toward parameters that already have small gradients.

Decoupled weight decay (AdamW) corrects this by applying λw directly to the parameters before the adaptive step, not inside the gradient. The update becomes:

w_{t+1} = w_t - η * (m_t / (sqrt(v_t) + ε) + λ * w_t)

Now the decay term avoids division by sqrt(v_t). Every parameter receives the same proportional shrinkage regardless of gradient history.

A quick experiment: train a small BERT on MLM with Adam + L2 versus AdamW, both at λ=0.01. Track per-layer weight norm decay. Under Adam, Layer 0 norms drop roughly 40% less than Layer 11. Under AdamW, the curve stays flat. The gap compounds over epochs — by step 100k the divergence appears in downstream task accuracy.

Rare, but possible. If you believe large-gradient parameters are "more important" and deserve preservation, vanilla Adam + L2 encodes that prior automatically. No case has been observed where this outperforms explicit importance weighting, yet the inductive bias exists.

If you use Adam and add λ * ||w||^2 to your loss, you are not doing weight decay — you are doing gradient-dependent decay. Switch to AdamW or implement decoupled decay manually. The one-line fix in PyTorch:

optimizer = torch.optim.AdamW(model.parameters(), lr=lr, weight_decay=wd)

Not Adam(..., weight_decay=wd). The argument name is identical; the behavior is not.