To understand the physically coupled cycles of water, energy, and carbon, simulating the Earth system at kilometer-scale resolution is now a critical technical requirement. Achieving this demands a throughput of at least 100 simulated days per day, a threshold pushing the limits of current high-performance computing. The German effort is the Icosahedral Nonhydrostatic (ICON) model; however, researchers suggest a fundamental shift in ICON’s development is needed to thrive on European supercomputers like JUPITER, with breakthroughs made possible by the Simple Cloud-Resolving E3SM Atmosphere Model (SCREAM) and a Kokkos-based approach.
ICON Achieves 1.25km Global Simulation with OpenACC on JUPITER
Taking full advantage of NVIDIA’s Grace Hopper superchips on both Alps and JUPITER, researchers achieved the first global Earth system simulation at 1.25 kilometer resolution, a landmark success enabled by adapting the ICON model for graphics processing unit acceleration using OpenACC compiler directives. This incremental port of the existing Fortran codebase was essential for maintaining both community acceptance and scientific integrity across ICON’s diverse modules, allowing data to remain resident on the GPU throughout the entire time-stepping loop to minimize costly memory transfers between the CPU and accelerator.
The resulting heterogeneous model achieved a temporal compression of 145.7 simulated days per day on the JUPITER system, demonstrating a significant increase in computational efficiency, the company says. Despite these achievements, the directive-based OpenACC method is approaching a point of diminishing returns, creating a portability bottleneck that limits ICON’s future development.
Currently, the OpenACC standard receives full support only from the nvhpc compiler on NVIDIA hardware, effectively preventing ICON from running on other architectures like AMD GPUs or Intel-based exascale systems such as Aurora, or limiting implementations to those with unresolved issues. This vendor lock-in poses a critical challenge to the model’s long-term viability as the high-performance computing landscape diversifies. The path forward for ICON now centers on embracing lessons from the Kokkos and domain-specific language approaches exemplified by the Simple Cloud-Resolving E3SM Atmosphere Model, or SCREAM.
“With the adoption of more sophisticated abstraction layers and the C++ and Kokkos-like programming models that are used so effectively by SCREAM and Omega, the WarmWorld project aims to deliver a model that is not only scientifically ‘alpha and omega,’ but also technologically resilient across the diverse landscape of global exascale computing,” according to the research. The WarmWorld project intends to finalize the transition of Earth system modeling from a discipline constrained by legacy code to a partnership between climate scientists and computer scientists, using abstraction layers to ensure performance portability.
Dipankar, A., Bianco, M., Bukenberger, M., Ehrengruber, T., Farabullini, N., Gopal, A., … Schulthess, T. report in Toward exascale climate modelling: A python DSL approach to ICON’s (Icosahedral Non-hydrostatic) dynamical core (icon-exclaim v0. 2. 0) that the team is pursuing this transition. Donahue, A.S., Caldwell, P.M., Bertagna, L., Beydoun, H., Bogenschutz, P.A., Bradley, A.M., … Zhang, Y.
Heterogeneous Computing Optimizes ICON’s Atmospheric and Land Components
Mapping computationally intensive atmospheric and land processes to graphics processing units (GPUs) while using central processing units (CPUs) for ocean and biogeochemistry components proved a crucial optimization strategy for the Icosahedral Nonhydrostatic (ICON) model. This heterogeneous approach, implemented on the JUPITER system, yielded a temporal compression of 145.7 SDPD, demonstrating a significant acceleration in modeling capabilities, according to the company.
Further refinement involved replacing ICON’s original dynamical core, previously built with OpenACC, with a new Data-Centric Parallel Programming (DaCe) implementation, enhancing performance through optimized data handling. NVIDIA’s Compute Unified Device Architecture (CUDA) graphs were also integrated to minimize latency within the land model’s smaller computational kernels.
To overcome this, the ICON community is actively exploring a paradigm, decoupling mathematical expressions from low-level data layout and instruction scheduling. The EXCLAIM project, for example, has refactored the atmospheric dynamical core using GT4Py, a Python-based domain-specific language, creating a more adaptable and architecture-agnostic codebase. This shift aims to facilitate portability across diverse high-performance computing (HPC) systems and prepare ICON for exascale architectures. Claudia Frauen, leading the ICON model group in application support, focuses on portability and performance to enable Earth-system simulations on modern HPC architectures.
Georgiana Mania, also in application support, specializes in graphics processing unit porting of the ICON model code, concentrating on performance-portable scientific software and accelerating large-scale climate-model components. Daniel Klocke, team lead of the Computational Infrastructure and Model Development group at the Max Planck Institute for Meteorology, coordinates the development of the ICON model, prioritizing high-resolution climate simulations and high performance computing.
The successful implementation of these heterogeneous computing strategies and the adoption of new software paradigms will be essential for ICON to not only meet but exceed these demands, enabling more accurate and detailed climate predictions and weather forecasting.
OpenACC’s Limitations Increase ICON Code Complexity and Vendor Lock-in
The ICON model’s dynamical core now contains thousands of lines of code where less than 50 percent describes the physics, with the remainder consisting of architecture-specific optimizations, pragmas, and macros. This environment creates a substantial maintenance burden, as any change to the mathematical formulation requires updates to multiple layers of description potentially unavailable to developers at the time. Such complexity hinders adaptation and introduces risk into the long-term development of the model, limiting the speed at which new scientific insights can be incorporated.
Kilometer-scale Earth system models are predominantly memory-bandwidth bound, requiring management of nearly one trillion physical-spatial degrees of freedom at 1.25-kilometer resolution, demanding at least eight tebibytes of memory. This sensitivity to low-level system parameters and memory layout makes adaptation to diverse hardware difficult within the current Fortran-plus-directives paradigm, often necessitating code duplication.
Kokkos Enables Performance Portability for SCREAM and Omega Models
These models demonstrated native portability across diverse hardware, running on both AMD and NVIDIA GPUs without code modification, a feat previously challenging for Fortran-based systems. This capability stems from Kokkos’ high-level abstractions for multi-dimensional arrays, called Kokkos::Views, and parallel loops, which compile to optimized code for various vendor-specific backends. The benefits extend beyond simply running on different architectures; Kokkos facilitates hierarchical parallelism, exposing thread teams that better utilize the throughput of exascale chips.
This approach minimizes the index bookkeeping that traditionally burdens Fortran implementations, a critical advantage when managing the immense computational demands of kilometer-scale Earth system models. At 1. Omega Ocean Model, using Kokkos, is 1.4 times faster than its Fortran-based predecessor ( MPAS-Ocean ) on CPUs, and significantly more efficient on a per-watt basis on GPUs.
The project team found Kokkos’ use of multi-dimensional data concepts, mirroring Fortran arrays with Kokkos::View, simplified development, the company says. The flexibility to use any C++ compiler, addressing a limitation of other frameworks, and seamless integration into ICON’s build system further supported its adoption. Her work centers on enabling these simulations on modern high-performance computing architectures.
The success of SCREAM, the first climate model to run on both AMD and NVIDIA GPUs with on-node performance portability, demonstrates that a non-Fortran, abstraction-layer-based approach is viable and even superior for exascale performance. This resilience to change is important, as the library handles portability, minimizing code changes when new supercomputing architectures are introduced, and safeguarding scientific investment.
See today’s quantum computing news on Quantum Zeitgeist for the latest breakthroughs in qubits, hardware, algorithms, and industry deals.
