AI Summaries Now Blend Reviews With Product Details Effectively

Multi-source opinion summarisation, integrating product metadata with reviews, demonstrably improves user engagement. A new benchmark dataset, M-OS-EVAL, assesses summaries across seven dimensions, revealing that factually enriched summaries achieve stronger alignment with human judgement, with a Spearman correlation of 0.74, exceeding prior methods.

The increasing volume of online product reviews presents a challenge for consumers seeking concise, informative summaries. Researchers are now investigating how large language models (LLMs) can synthesise not only subjective opinions from these reviews, but also integrate objective product data, such as specifications and ratings, to create more comprehensive summaries. Anuj Attri, Arnav Attri, and colleagues from the Indian Institute of Technology Bombay, working with researchers from Flipkart, detail their work in ‘LLMs as Architects and Critics for Multi-Source Opinion Summarization’. They present M-OS-EVAL, a new benchmark dataset designed to rigorously evaluate these multi-source opinion summaries across seven key dimensions, and demonstrate a significant preference for these enriched summaries in user studies, achieving a Spearman correlation of \r{ho} = 0.74 with human judgement.

Multi-Source Opinion Summarisation Enhances Product Evaluation and User Decision-Making

The proliferation of e-commerce necessitates effective methods for processing the substantial volume of customer reviews, significantly influencing purchasing decisions, yet comprehensive product evaluation requires broader data integration. Traditional opinion summarisation techniques typically focus solely on these reviews, hindering a complete understanding for potential buyers. Consequently, decision-making can be hampered by incomplete information, creating a need for more holistic summarisation strategies that combine diverse data sources.

Recent advances in large language models (LLMs) offer potential solutions for multi-source data integration, but their application to opinion summarisation remains largely unexplored, requiring dedicated benchmarks and evaluation metrics to assess their effectiveness in synthesising diverse data types like reviews and technical specifications. To address this gap, research focuses on developing methodologies that combine user opinions with objective product data, creating comprehensive summaries that facilitate informed decision-making and enhance user satisfaction by providing actionable insights. This involves not only integrating the data but also evaluating the quality of the resulting summaries across multiple dimensions, including relevance, coherence, and factual accuracy.

Multi-Source Opinion Summarisation (M-OS) represents an advancement beyond conventional opinion summarisation techniques, actively integrating objective product metadata – encompassing descriptions, features, specifications and ratings – alongside subjective customer reviews. This holistic approach aims to deliver comprehensive summaries that facilitate informed decision-making by presenting both factual details and experiential feedback, introducing M-OS-EVAL, a new benchmark dataset designed to rigorously evaluate multi-source opinion summaries across seven key dimensions: fluency, coherence, relevance, faithfulness, aspect coverage, sentiment consistency and specificity. The study demonstrates that M-OS significantly improves user engagement, with a user study revealing that 87% of participants express a preference for M-OS over traditional opinion summaries, indicating that the inclusion of objective data alongside subjective opinions enhances the perceived value and usefulness of the summaries.

Central to the success of M-OS is a carefully engineered prompting system designed to guide large language models (LLMs) in generating and evaluating these comprehensive summaries, employing two distinct types of prompts: M-OS-GEN prompts, which instruct the LLM to create the summaries, and M-OS-EVAL prompts, which are used to assess their quality. These prompts are meticulously crafted to emphasise clarity, balance, and structured coverage of key product aspects, prioritising the integration of both objective and subjective data to ensure summaries are not biased towards one perspective. Furthermore, the prompts encourage a comprehensive evaluation, considering multiple sources of information and employing a guided scoring mechanism to ensure consistency and objectivity, making the quality of the output directly dependent on the clarity and precision of the instructions given to the LLM.

To rigorously assess the effectiveness of M-OS, researchers conducted a large-scale user study involving 300 participants, employing a comparative evaluation where participants were presented with pairs of summaries – one generated using M-OS and the other a traditional opinion summary – and asked to evaluate them across several key dimensions. These dimensions included information comprehensiveness, decision confidence, specification understanding, research efficiency, and support for purchase decisions, revealing a strong preference for M-OS across all criteria with statistical analysis utilising a chi-square goodness-of-fit test. The results demonstrated a statistically significant preference, with Cramer’s V, a measure of effect size, reaching 0.72, indicating a substantial impact, and approximately 86.6% of participants favoured the M-OS summaries, highlighting their superior ability to provide informative and helpful resources.

The implementation of M-OS involved experimentation with a diverse range of both closed-source, such as GPT-4, and open-source LLMs, including Mistral-7B, Gemma, and Llama, conducted on clusters of NVIDIA A100 GPUs, utilising specific parameter settings – including top_k=25, top_p=0.95, temperature=0.2, and n=100 – to fine-tune the LLM outputs. The development of M-OS-EVAL, a benchmark dataset specifically designed for evaluating multi-source opinion summaries across seven key dimensions – fluency, coherence, relevance, faithfulness, aspect coverage, sentiment consistency, and specificity – further strengthens the methodology. The resulting Spearman correlation of ρ = 0.74 between M-OS-PROMPTS and human judgement demonstrates a strong alignment with human preferences, surpassing the performance of previous methodologies and solidifying M-OS as a promising advancement in the field of opinion summarisation.

Statistical analysis confirms the strength of this preference, with a Cramer’s V of 0.72 denoting a large effect size, while a chi-square test (χ² = 3126.83, df = 1, p <.001) strongly supports the rejection of the null hypothesis – that there is no preference between M-OS and traditional summaries. The researchers meticulously tuned LLM parameters, including temperature, top_k and top_p, to ensure deterministic and coherent outputs, and conducted 100 evaluations per summary to account for the inherent stochasticity of these models, demonstrating that carefully crafted prompts – M-OS-PROMPTS – achieve stronger alignment with human judgement, attaining an average Spearman correlation of ρ = 0.74, exceeding the performance of previous methodologies. The research utilises a range of both closed-source (GPT-4) and open-source LLMs (Mistral, Gemma, Llama, Vicuna, Zephyr, Qwen) and conducts experiments on NVIDIA A100 clusters, providing detailed implementation specifics to enhance reproducibility.

The success of M-OS stems from its integration of both objective product specifications and subjective customer feedback, creating a more holistic and informative resource for potential buyers, employing carefully engineered prompts, termed M-OS-GEN for summary generation and M-OS-EVAL for quality assessment, which effectively guide large language models (LLMs) in producing balanced and detailed summaries. M-OS-EVAL, in particular, utilises a structured assessment framework with quantifiable scoring, enhancing the robustness and reliability of the evaluation process, and the research establishes a new benchmark dataset, M-OS-EVAL, designed specifically for evaluating multi-source opinion summaries across seven key dimensions: fluency, coherence, relevance, faithfulness, aspect coverage, sentiment consistency, and specificity. This dataset addresses a critical gap in the field, providing a standardised means for assessing the quality of these summaries and facilitating further advancements in the area, with the strong correlation between M-OS-PROMPTS and human judgement, with an average Spearman correlation of ρ = 0.74, validating the effectiveness of the prompt engineering approach.

Future work should explore the adaptability of M-OS to diverse product categories and data sources, investigating the optimal strategies for integrating varying types of objective and subjective information, expanding the M-OS-EVAL dataset to encompass a wider range of products and user demographics to further enhance its generalisability and utility. Investigating the potential of M-OS to personalise summaries based on individual user preferences and needs represents another promising avenue for future research, and exploring the computational efficiency of M-OS and its scalability to large-scale e-commerce platforms is crucial for practical implementation, ensuring that this technology can be widely adopted and benefit both consumers and businesses.