OpenMC: The Open-Source Application Revolutionizing GPU-Based Supercomputing

OpenMC, an open-source Monte Carlo neutral particle transport application, has been adapted for Graphics Processing Units (GPUs) using the OpenMP target offloading model. It has been used to examine the performance of supercomputers like Frontier, Polaris, and Aurora. OpenMC’s significance lies in its ability to achieve performance portability across all three major GPU vendors: AMD, NVIDIA, and Intel. The application’s performance on GPUs is compared to the traditional CPU-based version and other CPU-based Monte Carlo particle transport applications, providing insights into the efficiency and scalability of GPU-based systems.

What is OpenMC and what is its significance in supercomputing?

OpenMC is an open-source Monte Carlo neutral particle transport application that has recently been adapted for use on Graphics Processing Units (GPUs) using the OpenMP target offloading model. This application has been used to examine the performance of supercomputers such as Frontier, Polaris, and Aurora. The significance of OpenMC lies in its ability to achieve performance portability across all three major GPU vendors: AMD, NVIDIA, and Intel.

The performance of OpenMC on GPUs is compared to both the traditional CPU-based version of OpenMC and several other state-of-the-art CPU-based Monte Carlo particle transport applications. This comparison provides valuable insights into the efficiency and scalability of GPU-based systems. The study also provides historical context by analyzing OpenMC’s performance on several legacy GPU and CPU architectures.

This research includes some of the first published results for a scientific simulation application at scale on a supercomputer featuring Intel’s Max series Ponte Vecchio GPUs. It is also one of the first demonstrations of a large scientific production application using the OpenMP target offloading model to achieve high performance on all three major GPU platforms.

How Does OpenMC Contribute to the Evolution of Supercomputing?

The recent decade has seen a significant increase in interest in using the Monte Carlo (MC) method for routine reactor design and engineering simulation work. The high computational cost of using the MC method for full core simulations makes these problems ideal use cases for exascale class supercomputers. Therefore, there is a great need for MC applications like OpenMC that can run and scale efficiently and in a portable manner on all types of modern GPU-based systems.

The world’s first exascale supercomputer, Oak Ridge National Laboratory’s Frontier, features MI250X GPUs from AMD. The second exascale computer, Argonne National Laboratory’s Aurora, features Intel Ponte Vecchio PVC Max 1550 GPUs. These first entries into the exascale era of supercomputing are notable in that they do not use NVIDIA GPUs. This sudden diversification in the GPU market has immediate implications for scientific simulation application development.

Given the near-universal requirement for simulation applications to run in a portable manner on GPU, there has been a push in recent years for legacy CPU-based applications to be ported to GPU and reoptimized to run efficiently on GPU architectures. MC neutral particle transport is one such field of simulation science. Many MC codes have seen decades of development and optimization for CPU-based systems, and their stochastic direct simulation approach makes them inherently difficult to optimize on most computer architectures.

What Makes OpenMC a Unique Application?

OpenMC is a continuous-energy MC particle transport application that has demonstrated excellent performance and scalability on CPU-based systems. In addition to being an open-source application, OpenMC also provides a host of advanced modeling and simulation capabilities including depletion, advanced geometry representations, on-the-fly Doppler broadening, and multigroup cross-section generation.

Recently, OpenMC has also been ported to GPU using the OpenMP target offloading programming model. This porting allows OpenMC to utilize GPUs from all three major GPU vendors as well as CPUs within a single code. This makes OpenMC a unique application in the field of scientific simulation.

In the present work, the performance of OpenMC is analyzed on a wide variety of GPU architectures including GPUs manufactured by NVIDIA, AMD, and Intel. Additionally, OpenMC’s MPI domain replication capabilities are utilized to examine its scalability on larger GPU-based supercomputers. This analysis provides valuable insights into the efficiency and scalability of GPU-based systems.

How Does OpenMC Compare to Other Applications?

Beyond the work on the OpenMP offloading port of OpenMC, there is a growing body of research porting MC particle transport applications to GPUs. For instance, Shift, PRAGMA, and Mercury all utilize CUDA to leverage NVIDIA GPUs. Shift has also been ported to the HIP programming model in order to utilize AMD GPUs. There have also been several recent efforts to port smaller portions of OpenMC into the CUDA language.

The present work regarding OpenMC differs from all these efforts in a number of ways. The most notable of which is the usage of the OpenMP target offload programming model allowing it to utilize GPUs from all three major GPU vendors as well as CPUs within a single code. This makes OpenMC a unique and versatile application in the field of scientific simulation.

What is the Future of OpenMC and GPU-based Supercomputing?

The diversification in the GPU market and the need for portability in scientific simulation applications have led to a crisis of portability for established GPU codes. There are a variety of portable GPU programming models, yet all feature differing levels of support across vendors and are at different stages of maturity.

Given this scenario, the future of OpenMC and GPU-based supercomputing lies in the development and optimization of portable models that are capable of providing full performance portability for HPC production simulation applications at scale. The success of OpenMC in achieving performance portability across all three major GPU vendors is a significant step in this direction.

The future of OpenMC also lies in its ability to bring previously impractical simulation problems within reach on GPU-based exascale systems. With its advanced modeling and simulation capabilities and its ability to run efficiently on a wide variety of GPU architectures, OpenMC is poised to play a significant role in the future of supercomputing.