Large Step Sizes Break Robust Machine Learning Training in Unexpected Ways

Training machine learning models to resist malicious data corruptions behaves completely differently from standard model training when engineers use large update step sizes, a mathematical study reveals. While standard optimization smoothly settles into stable curvature boundaries, robust training methods can experience persistent loss instability and fundamentally altered geometry.

Machine learning engineers rely heavily on adversarial training to build models that withstand deliberate perturbations or worst-case input noise. Training these robust systems typically involves solving a complex min-max optimization problem using deterministic gradient descent, an iterative mathematical method that adjusts model parameters to minimize errors against crafted worst-case examples. Traditional theoretical analyses of these optimization dynamics have largely focused on infinitesimal or exponentially small step sizes, treating training updates as continuous-time flows. However, practical implementations frequently employ large constant step sizes to accelerate learning, a regime that introduces erratic loss behavior and complex geometrical properties in standard training.

To understand how large steps alter robust optimization, researchers Yi Feng and Andrea Paudice from Aarhus University investigated the training dynamics of deterministic gradient descent across adversarial linear logistic regression and deep neural networks. Their findings are available as a preprint on the arXiv preprint server. The authors examined three core aspects of the training process, namely implicit bias, optimization error convergence, and local loss landscape sharpness.

Persistent Bias and Broken Optimization Guarantees

When analyzing adversarial linear logistic regression, the authors established that gradient descent with arbitrary constant step sizes converges toward a robust maximum margin direction that depends strictly on the chosen adversarial radius. This behavior diverges from standard training, which drives model parameters toward the standard maximum margin on unperturbed data. Although previous research identified this discrepancy, those earlier proofs required step sizes that were exponentially small with respect to data dimensionality. The new analysis proves that this implicit bias gap persists for arbitrarily large constant step sizes, operating at an improved mathematical rate.

The investigation also uncovered severe optimization hurdles unique to robust training frameworks. In standard logistic regression, large step sizes trigger an initial non-monotonic phase where losses fluctuate before transitioning into stable, monotonic descent, a mechanism that yields accelerated convergence rates. In contrast, the authors demonstrated that robust training features no such uniform phase transition. For any specified time budget and perturbation radius, researchers can construct datasets where gradient descent optimization error remains bounded below by a constant value, causing the loss to increase infinitely often across training steps.

Contrasting Sharpness Evolution in Neural Networks

Beyond linear models, the researchers conducted extensive empirical experiments on deep neural architectures, including multilayer perceptrons, convolutional neural networks, residual networks, and vision transformers trained on the CIFAR-10 and SVHN datasets. They tracked the evolution of loss sharpness, defined as the maximal eigenvalue of the parameter Hessian matrix evaluated on fixed adversarial examples. In standard neural network training, sharpness typically exhibits progressive sharpening before settling into an edge-of-stability regime where it oscillates around a threshold defined as two divided by the step size.

Under robust training regimes, however, sharpness follows a distinct two-stage dynamic. During the first stage, sharpness increases monotonically, mirroring standard training behavior. Once it peaks, rather than hovering near the standard boundary, sharpness enters a second stage characterized by oscillations accompanied by an overall decreasing trend over longer timescales. Theoretical analysis of an adversarial minimalist model confirmed that robust optimization dynamics converge to points with sharpness levels significantly lower than the standard boundary.

These theoretical and empirical insights demonstrate that standard assumptions about large step size optimization break down in robust machine learning architectures. The findings highlight the need for new analytical tools to characterize how worst-case defenses reshape neural network loss landscapes.

The study was posted to the arXiv preprint server on November 5, 2026.