LLMs Optimise GPU Kernels, Accelerating Performance on New Architectures.

The pursuit of enhanced computational performance consistently drives innovation in accelerator architectures, yet realising the full potential of these systems requires highly optimised code, a process traditionally demanding significant expertise and iterative refinement. Researchers now explore the application of large language models (LLMs) to automate aspects of this optimisation, effectively creating an autonomous agent capable of evolving code to maximise performance. Martin Andrews, Sam Witteveen, and colleagues detail their work in “GPU Kernel Scientist: An LLM-Driven Framework for Iterative Kernel Optimization”, presenting a methodology where an LLM iteratively refines accelerator kernels, generating and testing hypotheses based on existing code and broader GPU literature, and utilising observed timing data as feedback. Their system addresses the challenges posed by complex, less-documented architectures, such as the MI300, and offers a potential pathway to accelerate kernel optimisation, particularly where human expertise is limited.

Achieving peak performance on modern graphics processing units (GPUs) presents considerable challenges, particularly with rapidly evolving architectures such as the AMD MI300, where development tools and comprehensive documentation often lag behind hardware advancements. Researchers have developed an automated methodology, termed “GPU Kernel Scientist”, which leverages large language models (LLMs) to iteratively refine GPU kernels, addressing these complexities and broadening access to high-performance computing. This system operates through a multi-stage evolutionary process, beginning with the selection of promising prior kernel versions as a foundation for further refinement and subsequent optimisation.

The GPU Kernel Scientist generates hypotheses for optimisation experiments, drawing upon existing code and a broad understanding of GPU architecture, then autonomously implements these through code modification and rigorous evaluation. Performance feedback derives solely from observed timing data, enabling the system to learn and adapt without relying on manual intervention or expert guidance. A kernel in this context refers to a program executed on the GPU, and optimisation focuses on maximising its speed and efficiency.

Current optimisation efforts concentrate on two key areas: alleviating a single-wave global memory write bottleneck and improving the efficiency of scale caching, both critical for maximising throughput and minimising latency. The single-wave bottleneck arises from serialising the final write operation to global memory, severely limiting parallelism and hindering overall performance. Researchers explore several solutions, including employing atomic operations to allow concurrent writes to distinct memory regions, utilising a hybrid approach of caching scales directly in registers when the number of scales permits, or utilising a combination of vectorising scale application and caching scales directly in registers, if the number of scales permits. Global memory refers to the GPU’s main memory, accessible by all processing units, while registers are small, fast storage locations within each processing unit.

The system actively balances parallelism with correctness and memory efficiency, recognising that optimal performance requires careful consideration of these factors. Researchers explore vectorising scale application or caching scales directly in registers, if the number of scales permits, to further optimise performance. Vectorisation involves performing the same operation on multiple data elements simultaneously, increasing throughput.

Researchers are also investigating the potential for the system to autonomously discover novel optimisation strategies, beyond those documented in existing literature. Future work will concentrate on integrating the system with quantitative performance data from ongoing competitions, enabling a more rigorous evaluation of its effectiveness, and expanding the knowledge base of the LLM with more comprehensive GPU literature and optimisation techniques.

The research highlights the importance of profiling and benchmarking to determine the most effective solution for a given hardware configuration and workload. The system actively learns from performance data, iteratively refining the kernel code to achieve improved efficiency and scalability. The research presents the architectural design, operational workflow, and qualitative insights gained from the system, demonstrating the potential of LLM-driven agents to democratise and accelerate GPU kernel optimisation, particularly in environments with limited resources or rapidly evolving hardware.