Profolaf Streamlines Systematic Literature Reviews, Enhancing Efficiency and Reproducibility with Semi-Automation

Systematic reviews and mapping studies form the bedrock of evidence-based research, yet conducting them remains a significant challenge due to the intensive labour and time required. To address this, Martim Afonso, Nuno Saavedra, and Bruno Lourenço, from INESC-ID and the University of Lisbon, alongside Alexandra Mendes and João F. Ferreira from INESC TEC and the University of Porto, present ProfOlaf, a new semi-automated tool designed to dramatically streamline the systematic review process. ProfOlaf supports efficient article collection through iterative snowballing, incorporating human oversight to ensure accuracy, and then assists researchers in analysing content, extracting key themes, and responding to specific research questions. By intelligently combining automation with guided manual input, ProfOlaf promises to enhance both the speed and quality of systematic reviews, while also improving their reproducibility across diverse scientific disciplines.

LLM-Assisted Snowballing for Literature Reviews

This document details ProfOlaf, a tool designed to improve the efficiency, rigor, and reproducibility of systematic literature reviews (SLRs), particularly in software engineering. It combines iterative snowballing, a method of finding relevant papers through citations, with Large Language Model (LLM)-assisted analysis, while maintaining human oversight. Key features include snowballing, LLM integration for data extraction and topic modeling, and a human-in-the-loop approach for validation and quality control, supporting the entire SLR process from initial search to data extraction. LLM-generated summaries are generally accurate and well-structured, covering key points.

While LLMs show promise in topic modeling, human validation is recommended to address potential omissions and over-assignments. LLMs accurately identify programming languages used in reviewed papers, though occasional over-assignment may occur. ProfOlaf balances automation and control, improving efficiency, enhancing rigor and reproducibility, and addressing the growing volume of research. In essence, it empowers researchers to conduct high-quality systematic literature reviews more efficiently by leveraging LLMs while retaining crucial human oversight.

Automated Snowballing for Systematic Review Synthesis

Scientists developed ProfOlaf, a semi-automated tool designed to streamline systematic reviews while maintaining methodological rigor. The system employs an iterative snowballing process for article collection, beginning with an initial set of articles and expanding through successive searches with human filtering to ensure relevance and quality. This effectively balances automated expansion with expert oversight. ProfOlaf gathers bibliographic information and ranks venues to prioritize sources, then applies a multi-stage screening process assessing metadata, language, venue ranking, availability, and publication year.

Disagreements between raters are resolved through discussion, ensuring consistent application of inclusion and exclusion criteria, continuing with screening by title and full paper to refine the collection. This rigorous screening process, combining automated data gathering with expert judgment, establishes a high-quality foundation for subsequent analysis. The core of ProfOlaf’s analytical capability lies in its integration of large language models, which assist in extracting key topics and enabling users to query the model regarding content, significantly reducing the time required for in-depth analysis and enhancing the reproducibility and quality of systematic reviews.

ProfOlaf Streamlines Systematic Review Workflows

ProfOlaf is a new Python tool designed to streamline systematic reviews, employing a structured methodology to enhance both efficiency and the quality of research synthesis. The system utilizes an iterative snowballing process for article collection, combining automated retrieval of citations and references with human-in-the-loop filtering to ensure relevance, currently compiling bibliographic information from Google Scholar, Semantic Scholar, and DBLP, with a design that facilitates the easy integration of additional search platforms. Following article collection, ProfOlaf implements a metadata screening phase, allowing researchers to filter results based on criteria such as venue ranking, publication year, and language. To assist with venue ranking, the tool calculates cosine similarity to identify previously ranked venues similar to those being classified, and searches external databases like Scimago and CORE, presenting ranking information alongside similarity scores to support informed decisions about article relevance and quality.

The core of ProfOlaf’s methodology involves progressive manual screening, beginning with title review, then abstract assessment, and finally full-text analysis. The system presents article titles with direct URLs for easy access, and facilitates discrepancy identification between multiple reviewers, prompting collaborative discussion and consensus-building to ensure a rigorous and transparent review process. The tool’s architecture is designed to be extensible, allowing researchers to adapt the methodology and integrate new features as needed.

ProfOlaf Streamlines Systematic Review Methodology

ProfOlaf represents a significant advancement in the methodology of systematic reviews, addressing challenges related to efficiency, rigor, and reproducibility. The tool combines iterative snowballing techniques for comprehensive article collection with large language model-assisted analysis, enabling researchers to extract key topics and answer queries about research papers more effectively. By striking a balance between automation and human oversight, ProfOlaf streamlines the review process while maintaining methodological quality. The development of ProfOlaf offers a valuable resource, particularly in rapidly expanding fields like software engineering, where keeping pace with the volume of new research is increasingly difficult. While the tool enhances many aspects of systematic reviews, it is not a fully automated solution and requires continued researcher involvement. Future work may focus on expanding the tool’s capabilities and exploring its application across diverse research domains.