Reinforcement Learning Improves Accuracy of Partial Differential Equation Approximations.

Researchers developed a reinforcement learning framework to create closure models – approximations used in simplified partial differential equations (PDEs) – using synthetically generated data. This approach successfully improved the accuracy of coarse-grained PDE solutions for equations modelling phenomena like fluid dynamics and advection. Notably, models trained on complex, non-uniform equations generalised effectively to simpler, uniform cases, suggesting a pathway to efficient modelling even with limited observational data. The method offers a means to balance computational cost and accuracy in simulating complex systems.

Accurately modelling complex physical systems frequently necessitates solving partial differential equations (PDEs). However, fully resolving all relevant scales within these equations is often computationally intractable, forcing reliance on approximations that sacrifice detail. Researchers are now exploring machine learning techniques to improve these approximations, specifically through the development of ‘closure’ models which represent the effects of unresolved scales. A collaborative team, comprising Lothar Heimbach (ETH Zurich/Harvard University), Sebastian Kaltenbach, Petr Karnakov, Francis J. Alexander (Argonne National Laboratory), and Petros Koumoutsakos (Harvard University), detail a novel approach in their paper, “Reinforcement Learning Closures for Underresolved Partial Differential Equations using Synthetic Data”. They present a framework utilising reinforcement learning, trained on synthetically generated data, to create closure models for coarse-grained PDEs, demonstrating its effectiveness with equations governing fluid dynamics and showcasing the potential for generalisation across different system types.

Reinforcement Learning Enhances Coarse-Grained PDE Solvers

Partial differential equations (PDEs) model diverse physical and societal phenomena, yet obtaining accurate solutions often demands excessive computational resources due to the need to resolve fine-scale details. Consequently, practitioners frequently employ coarse-grained approximations, sacrificing accuracy for efficiency, and researchers continually seek methods to improve these approximations without incurring prohibitive costs. Recent research presents a novel framework utilizing reinforcement learning (RL) to develop closure models, enhancing the accuracy of coarse-grained PDE solvers and offering a promising pathway towards more efficient simulations.

The methodology centers on training an RL agent to correct the output of a coarse-grained numerical solver (CGS), effectively bridging the gap between computational efficiency and solution accuracy. This hybrid approach leverages the speed of the CGS while the RL agent learns to refine the solution, minimizing the error compared to a high-fidelity reference solution obtained through more computationally intensive methods. Data generated via the method of manufactured solutions provides the training set for the RL agent, allowing for controlled experimentation and evaluation of the framework’s performance across different PDE types.

Researchers demonstrate this framework using the one- and two-dimensional Burgers’ equations, and the two-dimensional advection equation, showcasing its versatility across different PDE types and validating its potential for broad applicability. The study highlights the superior generalisation capability of the RL-based closures, a critical advantage in real-world scenarios where encountering unseen conditions is common. Comparisons against a Fourier Neural Operator (FNO), a direct learning approach to PDE solutions, reveal that both methods achieve comparable accuracy on in-distribution test cases, but the RL closures significantly outperform the FNO when applied to out-of-distribution scenarios, specifically homogeneous PDEs with unseen initial conditions.

This improved generalisation stems from the RL agent’s ability to learn corrections based on the information already present in the CGS output, allowing it to adapt to new situations more effectively. The RL agent doesn’t simply memorize solutions, but rather learns a policy for correcting the CGS output, enabling it to extrapolate beyond the training data and handle unseen scenarios with greater accuracy. This adaptive learning capability is a key advantage of the RL approach, distinguishing it from more traditional methods that rely on fixed parameters or pre-computed solutions.

Crucially, the study demonstrates the potential for developing accurate and computationally efficient closure models, particularly in situations where data is scarce. By effectively bridging the gap between coarse-grained approximations and high-fidelity solutions, this research offers a promising avenue for advancing PDE modelling across a range of scientific and engineering disciplines. The framework’s ability to generalise to unseen scenarios positions it as a valuable tool for tackling complex problems where obtaining extensive training data is challenging or impossible.

The results demonstrate that the RL-based closure models achieve comparable accuracy to Fourier Neural Operators (FNOs) when applied to in-distribution test cases, confirming the effectiveness of the proposed approach. However, the RL approach surpasses the FNO in out-of-distribution generalisation, showcasing its ability to handle unseen scenarios with greater robustness. This improved performance stems from the RL agent’s ability to leverage the coarse-grained solution during prediction, focusing corrective actions on the existing approximation’s errors.

Researchers validate the framework using the one- and two-dimensional Burgers’ equations, alongside the two-dimensional advection equation, demonstrating its versatility across different PDE types and confirming its potential for broad applicability. Furthermore, the research establishes the capacity of closure models trained on inhomogeneous PDEs to generalise effectively to homogeneous counterparts, broadening the applicability of the methodology and offering a more flexible approach to PDE modelling. This transferability suggests a potential for developing versatile closure models applicable across a range of PDE problems, reducing the need for problem-specific training data.

The study highlights the potential for developing accurate and computationally efficient closure models, particularly in situations where data is scarce, offering a significant advantage in real-world applications. By effectively bridging the gap between coarse-grained approximations and high-fidelity solutions, this research offers a promising avenue for advancing PDE modelling across a range of scientific and engineering disciplines. The framework’s ability to generalise to unseen scenarios positions it as a valuable tool for tackling complex problems where obtaining extensive training data is challenging or impossible.

Future work should investigate the scalability of this approach to higher-dimensional problems and more complex PDE systems, expanding its applicability to a wider range of real-world applications. Exploring alternative RL algorithms and reward functions could further optimise the performance and efficiency of the closure models, potentially leading to even more accurate and robust solutions. A key area for development is the reduction of reliance on synthetic data, potentially through the incorporation of limited real-world observations or physics-informed learning techniques.

Quantifying the uncertainty associated with the closure models and incorporating this into predictive simulations represents a valuable direction for future research, enhancing the reliability and trustworthiness of the results. Investigating the use of transfer learning techniques could further improve the generalisation capability of the framework, allowing it to adapt to new problems with minimal retraining. Finally, exploring the potential for combining the RL approach with other machine learning techniques, such as deep learning, could lead to even more powerful and versatile PDE solvers.

This research demonstrates a promising new approach to solving PDEs, offering a potential pathway towards more efficient and accurate simulations. By leveraging the power of reinforcement learning, researchers have developed a framework that can adapt to new situations and generalize to unseen data, overcoming some of the limitations of traditional methods. The results of this study have significant implications for a wide range of scientific and engineering disciplines, and future work promises to further expand the capabilities of this innovative approach.

The framework’s ability to handle complex scenarios and generalize to unseen data makes it a valuable tool for tackling real-world problems where obtaining accurate solutions is critical. By bridging the gap between computational efficiency and solution accuracy, this research offers a promising pathway towards more efficient and reliable simulations, paving the way for new discoveries and innovations in a wide range of fields. The continued development of this approach promises to further expand its capabilities and solidify its position as a leading technique for solving PDEs.