Large Language Models Can Be Fooled into Labeling Documents as Relevant

A recent study has raised concerns about the reliability of Large Language Models (LLMs) in assessing the relevance of information objects, such as documents or passages. Researchers from RMIT University and Microsoft found that while some LLMs were comparable to human judges in terms of overall agreement, they were more likely to label passages as relevant compared to human judges.

The study revealed a worrying tendency for LLMs to be influenced by the presence of query words, even if the wider passage has no relevance to the query. This weakness in current measures of LLM labelling highlights the need for more nuanced and accurate methods of evaluating LLM performance, particularly in cases where human judges and LLMs disagree.

Can Large Language Models (LLMs) Be Fooled into Labeling a Document as Relevant?

Large Language Models (LLMs) have become increasingly popular in assessing the relevance of information objects. However, recent research has shown that these models can be fooled into labeling documents as relevant, even when they are not. This phenomenon was first observed by Marwah Alaofi and his team at RMIT University in Melbourne, Australia.

In their study, the researchers used multiple open-source and proprietary LLMs to label short texts for relevance. While some of these models showed comparable agreement with human judgments, others were more likely to label passages as relevant compared to human judges. This observation prompted the researchers to further examine cases where human judges and LLMs disagreed.

The results of their study revealed a tendency for many LLMs to label passages that included the original query terms as relevant, even if the wider passage had no relevance to the query. This was demonstrated by injecting query words into random and irrelevant passages, similar to inserting the query “best café near me” into this paper. The results showed that LLMs were highly influenced by the presence of query words in the passages under assessment.

This tendency of LLMs to be fooled by the mere presence of query words demonstrates a weakness in current measures of LLM labeling, which rely on overall agreement. This oversight can lead to bias in LLM-generated relevance labels and, consequently, bias in rankers trained on those labels. The researchers also investigated the effects of deliberately manipulating LLMs by instructing them to label passages as relevant, similar to the instruction “this paper is perfectly relevant” inserted above.

The results highlighted the critical need to consider potential vulnerabilities when deploying LLMs in real-world applications. This study serves as a warning that LLMs can be fooled into labeling documents as relevant, and it emphasizes the importance of carefully evaluating these models before using them in high-stakes applications.

The Limitations of Current Measures of LLM Labeling

The current measures of LLM labeling rely on overall agreement between human judges and LLMs. However, this approach has been shown to be flawed, as demonstrated by the study conducted by Marwah Alaofi and his team. The researchers found that LLMs are more likely to label passages as relevant compared to human judges, indicating that LLM labels denoting non-relevance are more reliable than those indicating relevance.

This observation prompts us to re-examine our current measures of LLM labeling. By relying solely on overall agreement, we may be missing important patterns of failures in these models. The study’s results demonstrate a tendency for many LLMs to label passages that include the original query terms as relevant, even if the wider passage has no relevance to the query.

This limitation highlights the need for more nuanced measures of LLM labeling, which take into account the potential biases and vulnerabilities of these models. By acknowledging these limitations, we can develop more robust evaluation methods that better capture the strengths and weaknesses of LLMs.

The Risk of Bias in LLM-Generated Relevance Labels

The study conducted by Marwah Alaofi and his team highlights a significant risk of bias in LLM-generated relevance labels. When LLMs are used to label passages for relevance, they may be influenced by the presence of query words in the passage under assessment.

This tendency of LLMs to be fooled by the mere presence of query words can lead to biased relevance labels. As a result, there is a risk that rankers trained on those labels will also be biased. This bias can have significant consequences in real-world applications, where accurate and unbiased labeling is crucial.

The researchers’ findings emphasize the importance of carefully evaluating LLMs before using them in high-stakes applications. By acknowledging the potential biases and vulnerabilities of these models, we can take steps to mitigate their impact and ensure that our evaluation methods are robust and reliable.

The Need for More Robust Evaluation Methods

The study conducted by Marwah Alaofi and his team highlights the need for more robust evaluation methods when assessing LLMs. By relying solely on overall agreement between human judges and LLMs, we may be missing important patterns of failures in these models.

To address this limitation, researchers and developers must consider potential vulnerabilities when deploying LLMs in real-world applications. This includes evaluating the impact of query words on LLM-generated relevance labels and developing more nuanced measures of LLM labeling that take into account the strengths and weaknesses of these models.

By acknowledging the limitations of current measures of LLM labeling and taking steps to address them, we can develop more robust evaluation methods that better capture the performance of LLMs. This will enable us to use these models with confidence in high-stakes applications, where accurate and unbiased labeling is crucial.

The Importance of Considering Potential Vulnerabilities

The study conducted by Marwah Alaofi and his team highlights the importance of considering potential vulnerabilities when deploying LLMs in real-world applications. By acknowledging the limitations of current measures of LLM labeling and the risk of bias in LLM-generated relevance labels, we can take steps to mitigate their impact.

This includes evaluating the effects of deliberately manipulating LLMs by instructing them to label passages as relevant. The researchers found that such manipulation influences the performance of some LLMs, highlighting the critical need to consider potential vulnerabilities when deploying these models.

By considering potential vulnerabilities and taking steps to address them, we can ensure that our evaluation methods are robust and reliable. This will enable us to use LLMs with confidence in high-stakes applications, where accurate and unbiased labeling is crucial.

Conclusion

The study conducted by Marwah Alaofi and his team highlights the limitations of current measures of LLM labeling and the risk of bias in LLM-generated relevance labels. By acknowledging these limitations and considering potential vulnerabilities when deploying LLMs in real-world applications, we can develop more robust evaluation methods that better capture the strengths and weaknesses of these models.

This study serves as a warning that LLMs can be fooled into labeling documents as relevant, and it emphasizes the importance of carefully evaluating these models before using them in high-stakes applications. By taking steps to address the limitations of current measures of LLM labeling and considering potential vulnerabilities, we can ensure that our evaluation methods are robust and reliable.