Large Language Models Achieve Fine-Grained Opinion Analysis with Reduced Human Effort

Researchers are tackling the significant challenge of creating detailed, labelled datasets for fine-grained opinion analysis, a crucial step in understanding nuanced sentiments expressed in text. Gaurav Negi, MA Waskow, and Paul Buitelaar from the Data Science Institute, University of Galway, Ireland, demonstrate a novel approach utilising large language models (LLMs) to automate both the annotation process and the resolution of differing labels. Their work addresses the considerable human effort and cost traditionally associated with building these datasets, particularly when applied across varied topics and real-world scenarios. By presenting a declarative annotation pipeline and a methodology for LLM-based adjudication, the team proves LLMs can function as reliable automatic annotators, achieving high levels of agreement and ultimately reducing the burden on human labelers.

Their work addresses the considerable human effort and cost traditionally associated with building these datasets, particularly when applied across varied topics and real-world scenarios. By presenting a declarative annotation pipeline and a methodology for LLM-based adjudication, the team proves LLMs can function as reliable automatic annotators, achieving high levels of agreement and ultimately reducing the burden on human labelers.

LLMs for Automated Opinion Annotation and Adjudication offer

Scientists have demonstrated a groundbreaking approach to fine-grained opinion analysis, leveraging the power of Large Language Models (LLMs) as both automatic annotators and adjudication systems. This research addresses a critical bottleneck in the field: the substantial cost and human effort required to create high-quality, labelled datasets for training models, particularly when dealing with diverse domains and real-world applications. The team achieved this by developing a declarative annotation pipeline, significantly reducing the variability inherent in manual prompt engineering when utilising LLMs to pinpoint precise opinion spans within text. Furthermore, they present a novel methodology enabling an LLM to effectively adjudicate multiple labels, ultimately producing highly consistent final annotations.
This study unveils a new paradigm for generating fine-grained opinion-annotated datasets, moving beyond reliance on expensive and time-consuming human annotation. Researchers employed a declarative pipeline, utilising LLMs to identify opinion spans and then implemented a unique adjudication process where a separate LLM resolves discrepancies between multiple initial annotations. Experiments focused on two key tasks: Aspect Sentiment Triplet Extraction (ASTE) and Aspect-Category-Opinion-Sentiment (ACOS) analysis, demonstrating the feasibility of this automated approach. The work establishes that LLMs can achieve high Inter-Annotator Agreement, indicating a level of consistency comparable to human annotators, thereby substantially reducing both cost and human workload.

The core innovation lies in the combination of a structured annotation pipeline with an LLM-based adjudication system. The declarative pipeline, built using DSPy, ensures consistent prompt preparation and reduces the impact of prompt variations on model performance. This approach moves away from ad-hoc prompt engineering towards a more programmatic and reliable method for harnessing LLM capabilities. The adjudication methodology then takes multiple LLM-generated annotations and synthesises them into a single, refined annotation, effectively resolving disagreements and improving overall data quality.

This process is particularly valuable for complex tasks like ASTE, which identifies the target, sentiment, and reason behind an opinion, and ACOS, which adds an aspect category for more nuanced analysis. Specifically, the research highlights the ability of LLMs to accurately extract aspect sentiment triplets, identifying what is being discussed, how the sentiment is expressed (positive, negative, or neutral), and why that sentiment is held. The ACOS task further expands on this by incorporating aspect categories, allowing for the analysis of implicit opinions where the aspect term is not explicitly stated. By successfully applying this methodology to both ASTE and ACOS, the team proves the versatility and effectiveness of their approach. This breakthrough opens exciting possibilities for real-world applications, including improved sentiment analysis in customer reviews, social media monitoring, and market research, all without the prohibitive costs associated with traditional human annotation.

LLM Annotation Pipeline and Adjudication for Opinion Analysis

Scientists investigated the feasibility of utilising Large Language Models (LLMs) as automatic annotators for fine-grained opinion analysis, addressing the critical shortage of labelled datasets, particularly within specialised domains. The research team engineered a declarative annotation pipeline to minimise inconsistencies inherent in manual prompt engineering when employing LLMs to pinpoint precise opinion spans within text. This pipeline leverages the instruction-following and reasoning capabilities of LLMs to streamline the annotation process and reduce reliance on extensive human effort. Crucially, the study pioneered a novel methodology for LLM-based adjudication, enabling a single LLM to synthesise multiple labels and generate consolidated, final annotations.

Experiments employed models of varying sizes to assess performance on both Aspect Sentiment Triplet Extraction (ASTE) and Aspect-Category-Opinion-Sentiment (ACOS) analysis tasks. The team implemented a system where multiple LLM annotators independently labelled text, and then a designated LLM adjudicator resolved any discrepancies, effectively mimicking the consensus-building process of human annotators. This approach demonstrably achieves high Inter-Annotator Agreement across the LLM-based annotators, indicating robust and reliable performance. The declarative pipeline utilises DSPy, a framework that allows researchers to define annotation tasks in a structured, programmatic manner, ensuring consistency and reproducibility.

Data was processed through this pipeline, with each LLM receiving the same input and instructions, thereby minimising variability introduced by differing prompt formulations. The adjudication process involved the LLM adjudicator receiving the annotations from all contributing LLMs and generating a final, unified annotation based on the collective input. This innovative method allows for the creation of high-quality, fine-grained opinion-annotated datasets with significantly reduced cost and human intervention. Furthermore, the research highlights the potential of LLMs to identify not only sentiment polarity but also the specific aspect terms and categories associated with expressed opinions. For instance, the system can accurately extract triplets like “hoped for better” as the opinion, “battery life” as the aspect, and assign it to the category “Battery#Operational_performance”. The team demonstrated that this approach surpasses the limitations of existing datasets, which are often small, domain-specific, and derived primarily from review websites, paving the way for more robust and generalisable sentiment analysis models.

LLMs excel at ASTE over ACOS annotation, demonstrating

Scientists have demonstrated that Large Language Models (LLMs) can function effectively as automatic annotators, significantly reducing the cost and effort associated with creating human-annotated datasets for fine-grained opinion analysis. The evaluation of these LLM-generated annotations revealed that ACOS tasks presented greater deviation from human annotations compared to ASTE tasks. Further investigation explored the challenges contributing to this difference, suggesting inherent complexities within the ACOS framework itself. Reliability of the annotated datasets was assessed using Kirppendorff α, confirming the viability of this approach across various model sizes and paving the way for more scalable and cost-effective dataset creation, either by assisting human annotators or functioning as independent annotators.

However, the authors acknowledge that ACOS tasks are more challenging for LLMs than ASTE tasks, indicating potential limitations in the current methodology when applied to more complex opinion analysis scenarios. Future research could focus on refining the LLM adjudication process or developing techniques to specifically address the difficulties inherent in ACOS analysis, potentially through improved prompting strategies or model architectures. This work represents a substantial step towards automating the creation of fine-grained opinion datasets, enabling broader applications in sentiment analysis and Natural language processing.