LLM Evaluation Achieves 90% Accuracy with New Define-Test-Diagnose-Fix Workflow

Researchers are tackling the complex problem of reliably evaluating Large Language Model (LLM) applications, which present unique challenges due to their stochastic nature and high-dimensional outputs. Daniel Commey from When, Better, Prompt, Evalua, Driven, Abstra, LLM, applic, high, dimens, alongside colleagues, propose a novel evaluation-driven workflow , Define, Test, Diagnose, Fix , to transform these difficulties into a repeatable process. Their work is significant because it introduces the Minimum Viable Evaluation Suite (MVES), offering a tiered approach to assessing general LLM applications, retrieval-augmented generation (RAG), and agentic workflows, and highlights how seemingly beneficial prompt improvements can inadvertently degrade performance on specific tasks, as demonstrated by their experiments with Llama 3 and Qwen 2.5.

The team achieved a repeatable engineering loop, systematically assess and refine LLM performance. This breakthrough moves beyond traditional software testing paradigms, acknowledging the high-dimensionality and inherent variability of LLM outputs.

The study synthesises existing evaluation methods, including automated checks, human rubrics, and the innovative use of LLMs as judges, while critically analysing the potential failure modes of the latter. Researchers conducted reproducible local experiments using Ollama, specifically with the Llama 3 8B Instruct and Qwen 2.5 7B Instruct models, to demonstrate the impact of prompt engineering on performance. These experiments revealed that a generic “improved” prompt template, while enhancing instruction-following, could detrimentally affect extraction pass rates, decreasing from 100% to 90% for Llama 3, and RAG compliance, dropping from 93.3% to 80% under the same conditions. This work establishes that seemingly beneficial prompt modifications can introduce trade-offs, highlighting the need for careful claim calibration and iterative evaluation rather than relying on universal prompt solutions.

The MVES framework provides a structured approach to decomposing complex tasks, such as RAG, into measurable components, allowing for targeted assessment of retrieval quality, factual accuracy, and overall system performance. Furthermore, the research details a robust protocol for evaluating LLM judges, identifying potential biases and outlining recommended guardrails to ensure reliable and unbiased assessments. The researchers also present detailed guidance on test set design, emphasising representativeness, edge case coverage, and systematic design to maximise statistical power and minimise overfitting. Through case studies, a customer support assistant, an internal knowledge base RAG bot, and a summarisation pipeline, the team demonstrates the practical application of their framework in real-world scenarios. This iterative process transforms these complexities into a repeatable engineering loop, ensuring consistent and reliable assessment. Researchers synthesised existing evaluation methods, including automated checks, human rubrics, and the innovative use of LLM-as-judge, while also meticulously documenting known failure modes of LLM judges, such as position bias and instruction leakage.

To demonstrate the efficacy of their approach, the study employed reproducible local experiments utilising the Ollama framework with both Llama 3 8B Instruct and Qwen 2.5 7B Instruct models. These experiments involved a controlled comparison of task-specific and generic prompt templates on structured evaluation suites. The team engineered small, structured test suites to quantify the impact of prompt template changes, revealing a trade-off between different behaviours. Specifically, replacing task-specific prompts with a generic “improved” template on the Llama 3 model resulted in a decrease in extraction pass rate from 100% to 90% and RAG compliance from 93.3% to 80%, despite an improvement in instruction-following.

This finding underscores the importance of evaluation-driven prompt iteration and careful claim calibration, rather than relying on universal prompt recipes. Furthermore, the research pioneered a detailed analysis of LLM-as-judge failure modes, identifying biases like verbosity and self-preference, which can significantly skew evaluation results. Actionable checklists were also provided for test set design, metric selection, and human evaluation rubrics, facilitating robust and reliable LLM assessment. The study highlights the importance of careful evaluation and claim calibration, demonstrating that seemingly beneficial changes, such as employing a generic template, can inadvertently degrade performance in specific areas.

Experiments using Llama 3 and Qwen 2.5 models revealed trade-offs between behaviours; a generic template improved instruction-following but reduced extraction pass rates and RAG compliance. The authors acknowledge that their experiments were conducted on relatively small, structured suites and that further investigation is needed to understand how these findings generalise to more complex scenarios. They advocate for continuous, evaluation-driven iteration as a more reliable approach than seeking universal solutions for LLM assessment. All materials, including test suites and results, are publicly available to facilitate reproducibility and further research in this area.