Adam vs SGD with momentum on a controlled quadratic

Adam and momentum SGD use different optimizer state to turn gradients into parameter updates. Momentum SGD accumulates a velocity-like buffer. Adam maintains moving averages of gradients and squared gradients, with bias correction. Their learning-rate numbers do not have identical effects, so a comparison needs explicit settings and a declared task.

This experiment minimizes a two-dimensional quadratic on CPU in float64. Both optimizers start at (4, 1) and receive 100 full-gradient updates on f(x,y) = (x² + 20y²)/2. We record objective values rather than claiming a speed benchmark.

Make the comparison reproducible

The selected learning rates are demonstration settings, not the winners of a comprehensive tuning search. We use momentum 0.9 for SGD and Adam's explicit beta and epsilon settings. There is no weight decay, minibatch noise or learning-rate schedule.

The verification environment is Python 3.12.14 and PyTorch 2.14.1+cpu with one CPU thread. No GPU is used. We do not measure elapsed time or memory, so neither can be inferred from the final objective. The manifest records package versions for reproduction.

Run equal gradient-update budgets

import torch
torch.set_num_threads(1)
for name in ['momentum_SGD', 'Adam']:
    point = torch.tensor([4., 1.], dtype=torch.float64, requires_grad=True)
    if name == 'momentum_SGD':
        optimizer = torch.optim.SGD([point], lr=.05, momentum=.9)
    else:
        optimizer = torch.optim.Adam([point], lr=.1, betas=(.9, .999), eps=1e-8)
    def objective():
        return (point[0]**2 + 20*point[1]**2)/2
    snapshots = {0: objective().item()}
    for step in range(1, 101):
        optimizer.zero_grad(set_to_none=True)
        loss = objective()
        loss.backward()
        optimizer.step()
        if step in [10, 50, 100]:
            snapshots[step] = objective().item()
    print(name, {step: round(value, 6) for step, value in snapshots.items()})
    print('final point:', [round(x, 6) for x in point.detach().tolist()])
    assert snapshots[100] < snapshots[0]
    assert torch.isfinite(point).all()

Checked output:

momentum_SGD {0: 18.0, 10: 1.080743, 50: 0.024934, 100: 0.000217}
final point: [-0.019938, 0.00135]
Adam {0: 18.0, 10: 4.587923, 50: 0.026668, 100: 0.000316}
final point: [-0.02143, 0.002937]

Explain the trajectory instead of crowning a winner

The optimizers can overshoot or oscillate while still making progress overall. Momentum carries information across updates, while Adam adapts coordinate-wise update scales using gradient history. A point's current gradient alone does not determine its next update once the optimizer has state.

Different choices of learning rate and update budget can change which run ends with the lower objective. Equal update counts do not imply equal computational cost, and a simple convex quadratic does not represent the loss landscape or generalization behavior of a trained neural network.

Adam's state includes two moment estimates per parameter in this basic configuration; momentum SGD stores a momentum buffer. Actual memory also depends on parameter precision, gradients, implementation options and any distributed or mixed-precision machinery. This script does not measure those totals.

Separate optimization from generalization

This objective has a known minimum and no train/test split because it is a mathematical optimization demonstration. A prediction model needs held-out evaluation. Faster training-loss reduction does not establish better held-out accuracy, robustness or calibration.

For a practical comparison, hold the data split, model initialization, batch order and evaluation procedure fixed, tune each optimizer on development data with comparable budgets, and repeat over declared seeds. Record schedules, decay conventions and stopping rules. Do not use the final test score to select the optimizer.

Extend the controlled run

Change the curvature coefficient from 20 to 100 and rerun both configurations. Then tune each learning rate over a stated small development grid. Compare final objective and the full trajectory; report the grid rather than only the most favorable run.

The Deep Learning syllabus includes optimization and training; lessons are being published. Start with gradient descent stability to derive the plain-update boundary, and a forward/backward pass to verify the gradients before changing optimizers.

References

PyTorch optimizer documentation — Update definitions, defaults and implementation details for SGD and Adam.

Adam paper — Kingma and Ba describe adaptive moment estimation.