Large Language Models Optimise Code Directly From Compiler Reports.

CompilerGPT automatically improves code performance by interpreting compiler optimisation reports and utilising large language models. Experiments with GPT-4o and Claude Sonnet, alongside Clang and GCC compilers, achieved speedups up to 6.5x across five benchmark codes, suggesting a pathway to enhanced compiler usability and software optimisation.

The optimisation of computer code remains a critical, yet often laborious, task for programmers. Identifying performance bottlenecks and translating complex compiler feedback into effective code modifications requires significant expertise. Researchers are now exploring the application of large language models (LLMs) to automate aspects of this process. A team including Peter Pirkelbauer and Chunhua Liao, from Lawrence Livermore National Laboratory, detail their work in a paper titled ‘CompilerGPT: Leveraging Large Language Models for Analysing and Acting on Compiler Optimisation Reports’. They present a novel framework, CompilerGPT, which integrates compilers, LLMs and automated testing to interpret optimisation reports and directly modify code, achieving speed improvements of up to 6.5x in preliminary tests.

LLMs Automate Compiler Optimisation with CompilerGPT

Optimising compiler performance remains a complex task in software development, frequently requiring specialist knowledge to interpret diagnostic reports and manually rewrite code for enhanced speed. Recent research details CompilerGPT, a framework utilising large language models (LLMs) to automate this process, addressing the need for more accessible and efficient optimisation techniques. The system operates through iterative cycles of compilation, LLM-driven code rewriting, and performance evaluation, integrating compilers – specifically Clang and GCC – with LLMs and employing user-defined test harnesses to quantify optimisation impact.

CompilerGPT bridges the gap between technical diagnostics and practical code improvement, enabling the LLM to analyse compiler reports and propose code modifications aimed at enhancing performance. Experiments utilising both GPT-4o and Claude Sonnet demonstrate the potential of this approach, achieving speedups of up to 6.5x across five benchmark codes. These included established tests from the NAS Parallel Benchmarks – a suite used to evaluate high-performance computing systems – and the Smith-Waterman algorithm, commonly used in bioinformatics for sequence alignment. While performance gains were not consistently observed across all tests, the results indicate a significant capacity for LLMs to identify and implement effective optimisations, highlighting the framework’s adaptability.

This research represents an advance in applying LLMs to software engineering, streamlining optimisation by automating the interpretation of compiler feedback and subsequent code modification. The system relies on established benchmarks and parallel programming standards, such as OpenMP – an application programming interface that supports shared-memory parallel programming – ensuring relevance to contemporary software development practices. This suggests a future where LLMs actively assist developers in achieving optimal software performance, potentially broadening access to advanced optimisation techniques. The framework’s ability to process reports from different compilers underscores its versatility and potential for widespread adoption.

The LLM actively processes compiler feedback to refine existing code, suggesting a new paradigm for compiler usability. The framework’s iterative approach allows for repeated cycles of compilation, analysis, and modification, enabling a degree of automated refinement previously unattainable. Prompt engineering – the careful design of instructions given to the LLM – proves critical, with the specificity and quality of prompts directly influencing the LLM’s ability to identify meaningful optimisation opportunities. While the LLM demonstrates an aptitude for understanding assembly code semantics – the meaning of instructions in a low-level programming language – human verification remains essential to confirm the correctness and effectiveness of suggested changes, functioning as an assistive tool augmenting developer capabilities.

Despite the observed performance improvements, the research acknowledges limitations. Speedups were not consistently achieved across all benchmarks, indicating that the LLM’s effectiveness varies depending on the code’s complexity and characteristics. Further investigation is needed to understand the factors influencing the LLM’s performance and to develop strategies for improving its reliability. Future work should focus on expanding the range of benchmarks used to evaluate the framework, exploring different LLM architectures and prompting techniques, and investigating methods for automatically verifying the correctness of the generated code.