Machine Learning Uses Physics Rules As Extra Data to Improve Accuracy

Researchers are increasingly applying machine learning techniques to solve complex problems in computational physics, yet understanding the underlying statistical properties of these methods remains a significant challenge. David Barajas-Solano from Pacific Northwest National Laboratory, alongside colleagues, present a novel statistical learning analysis of physics-informed neural networks (PINNs) used for initial and boundary value problems. This work reformulates PINN parameter estimation as a statistical learning problem, revealing that the physics penalty acts as an infinite source of indirect data and the learning process minimises distributional divergence. The analysis demonstrates that physics-informed learning with PINNs constitutes a singular learning problem, employing tools such as the Local Learning Coefficient to assess parameter estimates for a heat equation IBVP. Ultimately, this research offers crucial insights into quantifying predictive uncertainty and improving the extrapolation capabilities of PINNs, advancing the field of scientific machine learning.

This work presents a novel statistical learning analysis of PINNs, moving beyond conventional interpretations of how these networks learn physical laws. Researchers have successfully reformulated the PINN learning process as a statistical learning problem, revealing that the ‘physics penalty’ traditionally viewed as a regularizing term, functions instead as an infinite source of indirect data. This insight fundamentally alters how we perceive the information PINNs utilise during training. The study focuses on initial and boundary value problems (IBVPs), common in fields like fluid dynamics and heat transfer, where finding exact solutions is often intractable. This analysis demonstrates that learning with PINNs is a ‘singular learning problem’, a characteristic of many deep learning models where traditional statistical tools fall short. The core of their approach lies in minimising the Kullback-Leibler divergence, a measure of how different two probability distributions are, between the true data-generating distribution and the distribution predicted by the PINN. This reformulation allows for a more precise quantification of predictive uncertainty and offers insights into the extrapolation capabilities of PINN models. By understanding the underlying statistical properties of PINN training, scientists can develop more robust and reliable models for a wide range of scientific and engineering applications. The implications extend to improved design of PINN architectures and optimisation strategies, potentially unlocking greater accuracy and efficiency in solving complex physical simulations. A statistical learning perspective underpinned the investigation into PINNs applied to IBVPs. The work reframed the estimation of PINN parameters as a problem of statistical learning. Rather than viewing the physics penalty as simple regularization, the research demonstrated its function as an infinite source of indirect data, effectively enriching the learning process. This conceptual shift enabled the application of tools from singular learning theory to analyse the behaviour of PINN parameter estimates obtained through stochastic optimisation for a heat equation IBVP. Specifically, the LLC, a key concept in singular learning, was employed to characterise the learning dynamics and identify potential challenges in parameter estimation. This approach focuses on the geometry of the loss landscape and the conditioning of the optimisation problem. The study deliberately incorporated hard constraints for initial and boundary conditions to facilitate the statistical re-formulation and provide a well-defined framework for analysis. By treating the physics penalty as data, the research team could then assess how effectively the PINN distribution of residuals aligns with the true underlying data-generating distribution, quantified using the Kullback-Leibler divergence. This divergence measure provides a means to evaluate the quality of the learned approximation and its sensitivity to the imposed constraints. Furthermore, this methodology offers insights into quantifying predictive uncertainty and assessing the extrapolation capabilities of PINNs, crucial aspects for reliable application in scientific modelling and data assimilation. The chosen approach moves beyond simply achieving accurate solutions on training data, instead focusing on the broader statistical properties of the learned model and its ability to generalise to unseen scenarios. This value was determined through analysis of the heat equation IBVP, irrespective of variations in batch size or learning rate. Specifically, experiments utilising batch sizes of 8, 16, and 32 all yielded LLC values clustered around 9.5, as evidenced by the computational results summarised in Figure 2. The study meticulously tracked training and test losses every 100 iterations, alongside LLC estimates computed every 10,000 iterations, to arrive at this consistent finding. Initial investigations employing a small value of σ = 1 × 10−2 resulted in negative LLC estimates and were therefore discounted. However, LLC values calculated with σ = 1 × 10−1 were consistently, though slightly, larger than those obtained with σ = 1, suggesting that PINN models adhering more closely to the underlying physics exhibit marginally greater complexity. Despite this small difference, the conservative choice of σ = 1 × 10−1 was adopted for further analysis. The observed consistency in LLC estimates is remarkable given the substantial differences in the local minima w⋆ across various experiments. Notably, the estimated LLC of 9.5 is significantly smaller than the total number of parameters in the PINN model, which stands at 20,601. This disparity indicates that the PINN solution resides within a remarkably flat region of the parameter space, meaning that diverse initializations and stochastic optimisation hyperparameters converge to the same region. A lower learning rate of 1 × 10−4, due to reduced stochastic optimisation noise, required a greater number of iterations to reach this region compared to the higher learning rate of 1 × 10−3, as demonstrated in Figures 2 and 3. Scientists are increasingly viewing PINNs not merely as clever function approximators but as statistical learning problems with unique characteristics. For years, the promise of PINNs has been to bridge the gap between data-scarce scientific modelling and the power of machine learning, but realising this potential has been hampered by difficulties in training and ensuring reliable predictions. This work offers a fresh perspective, reframing the ‘physics penalty’ within PINNs, traditionally seen as a regularisation technique, as a form of indirect data. The implications extend beyond simply improving training stability. By applying tools from singular learning theory, researchers have identified PINNs as facing a particularly challenging learning landscape, one where standard optimisation techniques can struggle. Understanding this ‘singularity’ is crucial for developing better algorithms and, importantly, for quantifying the uncertainty in PINN predictions. Current methods often provide overconfident results, a dangerous flaw when applied to critical systems. However, the focus on a relatively simple heat equation leaves open the question of how these findings generalise to more complex, high-dimensional problems. Future work will likely explore how to adapt these statistical learning tools to handle the intricacies of real-world physics, and whether similar principles can be applied to other physics-informed machine learning approaches, potentially unlocking a new era of robust and trustworthy scientific AI.