Sakuraone HPC Cluster Ranks Among World’s Top Computing Resources

The increasing demand for computational resources to support advanced artificial intelligence, particularly the training of large language models, necessitates innovative approaches to high-performance computing (HPC) infrastructure. Recent developments demonstrate the potential of privately funded HPC systems to contribute significantly to this landscape. Fumikazu Konishi, alongside colleagues at the SAKURA Internet Research Center, detail the development and performance of SAKURAONE, a managed HPC cluster designed to facilitate advanced workloads. Their work, entitled ‘SAKURAONE: Empowering Transparent and Open AI Platforms through Private-Sector HPC Investment in Japan’, showcases a system achieving notable performance and, crucially, employing a fully open networking stack based on 800 Gigabit Ethernet and the SONiC operating system, a configuration currently unique within the top 100 systems on the TOP500 list.

The SAKURAONE cluster attained a ranking of 49th globally based on the High Performance Linpack (HPL) benchmark, achieving a sustained performance of 33.95 PetaFLOPS/s (PFLOP/s) on this metric. Further evaluation using the High Performance Conjugate Gradient (HPCG) benchmark yielded 396.295 TeraFLOPS/s (TFLOP/s), while the HPL-MxP benchmark, designed to assess performance on low-precision workloads relevant to AI, demonstrated 339.86 PFLOP/s utilising FP8 precision. The system comprises 100 compute nodes, each equipped with eight H100 GPUs, and is supported by a 2 Petabyte all-flash Lustre storage subsystem. Inter-node communication benefits from a full-bisection bandwidth interconnect based on a Rail-Optimized topology, utilising 800 GbE links and RoCEv2 (RDMA over Converged Ethernet version 2) to facilitate high-speed, lossless data transfers.
Sakura currently ranks 49th globally in high-performance computing (HPC), as measured by the High Performance Linpack (HPL) benchmark, a widely used metric for assessing a system’s floating-point performance. The system delivers a sustained performance of 33.95 petaflops per second on HPL and 396.295 teraflops per second on the High Performance Conjugate Gradient (HPCG) benchmark, a test designed to better evaluate systems handling real-world scientific applications. These results confirm its capability to manage complex scientific simulations and data analysis tasks effectively, establishing it as a significant HPC resource.

A key differentiator for Sakura resides in its adoption of a fully open networking stack, utilising 800 Gigabit Ethernet (GbE) and the Software for Open Networking in the Cloud (SONiC) operating system. This represents a departure from traditional approaches that rely on proprietary technologies like InfiniBand, a high-bandwidth network communication standard, and validates the feasibility of open, vendor-neutral solutions for large-scale HPC infrastructure. This innovative approach encourages wider adoption of open standards within the HPC sector.

Sakura’s architecture comprises 100 compute nodes, each equipped with eight NVIDIA H100 GPUs, providing substantial processing capacity for both traditional HPC and emerging artificial intelligence (AI) workloads. The all-flash Lustre storage subsystem, with a capacity of 2 petabytes (equivalent to 2,000 terabytes), ensures high-throughput and low-latency data access, critical for data-intensive applications and efficient data management. This combination of processing and storage capabilities supports the training of large language models (LLMs) and other computationally demanding AI tasks, accelerating scientific discovery.

Furthermore, the system’s Rail-Optimized topology, utilising a full-bisection bandwidth interconnect with 800GbE links, facilitates high-speed, lossless data transfers between nodes. This topology, combined with RDMA over Converged Ethernet version 2 (RoCEv2), a protocol enabling efficient data transfer directly between application memory, mitigates communication bottlenecks and enhances performance in large-scale parallel applications, ensuring efficient data exchange. The design prioritises efficient data movement, crucial for maintaining performance as the system scales and handles increasingly complex workloads.

Sakura’s successful implementation of Ethernet, rather than the traditionally favoured InfiniBand, represents a significant contribution, demonstrating a viable alternative for high-performance networking in demanding computational environments and potentially lowering costs associated with proprietary interconnect technologies. This shift towards open standards fosters innovation and competition within the HPC ecosystem.

Sakura’s innovative approach to networking, utilising 800 Gigabit Ethernet and SONiC, distinguishes it from traditional HPC systems reliant on InfiniBand. This open networking stack offers several advantages, including lower costs, increased flexibility, and greater vendor choice. By embracing open standards, Sakura promotes innovation and collaboration within the HPC community, positioning it as a leader in the next generation of HPC infrastructure.