New Math Method Cuts AI Training Crashes by Ninefold in Complex Systems

When mathematicians build computer models to simulate complex systems like the spread of an epidemic or the flow of ocean currents, they often run into a frustrating bottleneck known as the closure problem. This happens when governing equations contain unknown terms representing hidden physical effects that are too small or complicated to track directly, such as sudden shifts in public quarantine behavior. To make the model work, researchers rely on closure models, which act as mathematical stand-ins for these missing pieces. A team of researchers at Sandia National Laboratories investigated how best to train these neural network closures using machine learning, comparing two very different strategies in a study available on the arXiv preprint server.

Why Training Neural Networks Inside Dynamic Simulators Breaks Down

The standard way to build a data-driven closure model is known as embedded training. In this approach, the neural network is placed directly inside the main simulation equations while the computer adjusts its weights. As the network tries out different parameters, it alters the system dynamics on the fly. While this method targets the exact trajectory of the system, it comes with a massive computational price tag and severe numerical fragility.

Every single time the model evaluates its training loss, it must run a full forward simulation of the system. Furthermore, optimization algorithms often need to differentiate through the numerical time steps, which can cause the simulation to crash. In the team’s experiments involving a synthetic epidemiological model with 2 million individuals, embedded training suffered a roughly forty percent failure rate. The network parameters frequently generated nonphysical states, mathematical stiffness, or exploding trajectories that caused the solver to fail entirely.

How Decoupling Data Assimilation From Training Avoids Crashes

To bypass these stability traps, the authors examined a staged approach that separates data assimilation from neural network training. Instead of forcing the neural network to sit inside the active simulation, the first stage uses a statistical tool called an ensemble Kalman smoother to look at noisy observations and infer a smooth, time-dependent trajectory for the missing closure term. This step estimates both the physical state and the hidden quarantine rates without needing a neural network.

Once the system has reconstructed these target trajectories, the second stage trains the neural network completely offline as a standard supervised regression problem. Because the simulator is no longer being called during every single gradient update, unstable intermediate network guesses cannot crash the solver. Across fifty independent test runs, this staged workflow eliminated solver failures entirely, achieving zero failed training runs compared to the hundreds of failures seen in embedded training.

What the Accuracy and Cost Comparison Reveals

The study found a clear trade-off between raw accuracy and computational efficiency. On average, the embedded approach achieved lower held-out test errors and more precise calibration closure accuracy. However, the staged approach managed to capture the broad qualitative behavior of the system with errors that remained within the same order of magnitude.

The efficiency gains for the staged method were substantial. Embedded training required an average of 24,100 simulator evaluations per run, whereas the staged method required only about 2,780 evaluations, marking an approximately ninefold reduction in computational cost. Most of the staged workflow’s computational budget was spent on maximum likelihood estimation for process noise during the initial data assimilation phase.

How Well Do These Learned Closures Generalize to New Scenarios

To see how robust the trained models were, the authors embedded the resulting closures into systems with entirely different disease transmission rates and recovery parameters. Both methods showed resilience when facing scenarios that deviated from their initial calibration data. When tested against stronger extrapolation scenarios, such as an accelerated recovery rate, both models experienced higher errors, but the staged approach showed greater replicate-to-replicate variability.

The authors conclude that whether a researcher should choose embedded or staged training depends entirely on the computational setting. For inexpensive, highly stable models, the extra accuracy of embedded training may be worth the computational strain. But for massive, expensive, or numerically fragile simulators where solver crashes are common, the dramatic reduction in model evaluations and flawless robustness offered by staged training provide a compelling alternative.

The study was posted to the arXiv preprint server on November 2, 2026.