LLMs Meet Regulatory Compliance: The DRAFT Approach

A new study published on May 2, 2025, titled Document Retrieval Augmented Fine-Tuning (DRAFT) for safety-critical software assessments, introduces an innovative method to enhance large language models (LLMs) for compliance with complex regulatory frameworks in software engineering. The research, conducted by a team of seven researchers, presents DRAFT as a dual-retrieval architecture that improves the accuracy and transparency of safety-critical software evaluations, achieving a 7% improvement in correctness over baseline models when tested with GPT-4o-mini.

Safety-critical software compliance assessment is traditionally constrained by manual evaluation under complex regulations. This paper introduces DRAFT (Document Retrieval-Augmented Fine-Tuning), enhancing large language models for compliance tasks. By integrating dual-retrieval architecture accessing both software documentation and reference standards, DRAFT improves upon existing RAG techniques. A semi-automated dataset generation method with distractors mimics real-world scenarios, enabling fine-tuning. Testing with GPT-4o-mini shows a 7% correctness improvement, better evidence handling, structured responses, and domain-specific reasoning. DRAFT offers a practical approach to compliance systems while maintaining transparency and evidence-based reasoning essential for regulatory domains.

In recent years, artificial intelligence has witnessed a transformative shift with the advent of Retrieval-Augmented Generation (RAG). This innovative approach is redefining how large language models (LLMs) operate, offering solutions to challenges that traditional models have struggled with. RAG’s impact is profound, promising significant advancements across various industries by enhancing adaptability and efficiency. At its core, RAG integrates retrieval mechanisms with generative models, enabling access to external information during text generation. This method allows LLMS to reference relevant data sources in real-time, producing contextually accurate responses. Unlike traditional methods that rely solely on pre-training data, RAG dynamically incorporates new information, making it highly adaptable and efficient.

RAG offers several advantages over conventional fine-tuning. Its efficiency reduces the need for extensive retraining by leveraging external knowledge bases. This adaptability allows models to quickly adjust to diverse domains without significant computational overhead. Additionally, RAG utilises existing data more effectively, minimising the requirement for large new datasets and enhancing data efficiency.

The versatility of RAG is evident across various sectors. In healthcare, it assists in medical decision-making by referencing clinical guidelines and patient records, as seen in applications for adolescent scoliosis management. Within cybersecurity, RAG enhances threat detection in transit systems by analysing patterns and alerts in real-time. Furthermore, in content generation, it supports the creation of accurate and relevant content across different domains, as highlighted in recent surveys.

Despite its benefits, RAG presents challenges that require careful consideration. Ensuring models maintain ethical standards when accessing external data is crucial, with recent studies emphasising the importance of safety alignment during adaptation. Additionally, the dynamic use of information raises questions about privacy and bias, necessitating robust safeguards to address these ethical concerns.

Retrieval-Augmented Generation represents a significant leap forward in LLM capabilities, offering efficient adaptability across various applications. While challenges remain, ongoing research addresses these issues, paving the way for safer and more effective AI systems. As RAG continues to evolve, its transformative impact is set to revolutionise industries, solving problems that traditional models couldn’t handle as effectively. The future of AI looks promising with RAG at the forefront, driving innovation and efficiency in unprecedented ways.