Stanford HAI Shows LLMs Aid Social Science Research

Research utilising large language models (LLMs) offers a potential means of simulating human data for social science investigations, encompassing fields such as economics, psychology, sociology, and political science. Jacy Anthis, a visiting scholar at the Stanford Institute for Human-Centered AI and PhD candidate at the University of Chicago, suggests LLMs can be employed to test assumptions, conduct pilot studies, and estimate sample sizes, thereby leveraging increased statistical power. While acknowledging limitations – including tendencies towards less varied, biased, or overly agreeable responses and poor generalisation to novel contexts – Anthis proposes that approximately one further year of research could yield significant advancements in this methodology. This approach aims to address the inherent complexities of social science research, which is complicated by its human subjects and often proves time-consuming, expensive, and difficult to replicate.

The Promise of AI in Social Science

The application of artificial intelligence, specifically large language models (LLMs), offers potential advancements in social science research, which traditionally encompasses fields such as economics, psychology, sociology, and political science. These models, designed to emulate human speech, can simulate human data, addressing inherent complexities in studying human subjects – a process often time-consuming, expensive, and difficult to replicate. LLMs allow researchers to inexpensively test assumptions, conduct pilot studies, and estimate appropriate sample sizes, effectively augmenting traditional methodologies like fieldwork, polling, and randomised controlled trials.
The integration of LLMs into the research pipeline offers the possibility of leveraging increased statistical power through the combination of human and AI-generated subjects. However, limitations exist; current LLMs can produce responses that are less varied, exhibit bias, or demonstrate a tendency towards agreement, and may not generalise effectively to novel situations. Despite these constraints, preliminary results suggest the viability of these ‘rough-and-ready’ methods, prompting researchers like Jacy Anthis, a visiting scholar at the Stanford Institute for Human-Centered AI, to advocate for further investigation, suggesting that a year of focused effort could yield significant progress in this area. Anthis’s work, detailed in a recent preprint article, highlights the potential for AI to address longstanding challenges in social science research.

Limitations of Language Models

Despite the potential benefits of integrating large language models (LLMs) into social science research, certain limitations currently impede their complete substitution for human subjects. Research indicates that LLMs frequently generate responses exhibiting reduced variability compared to human participants, potentially skewing results and limiting the breadth of insights obtained. Furthermore, these models are susceptible to inherent biases, which can manifest in their outputs and compromise the objectivity of the research. A notable characteristic identified is a tendency towards ‘sycophantic’ answers – a predisposition to agree with prompts or express favourable opinions – which introduces a systematic error distinct from genuine human responses.

Crucially, LLMs demonstrate limited capacity for generalisation; their performance deteriorates when applied to new or unfamiliar settings, hindering their utility in research contexts requiring adaptability and external validity. This lack of transferability restricts the scope of investigations and necessitates careful consideration of the environments to which LLM-generated data can be reliably applied. While acknowledging these constraints, researchers, including Jacy Anthis at the Stanford Institute for Human-Centered AI, remain optimistic, suggesting that focused development could mitigate these issues and enhance the reliability of LLMs as research tools. His recent preprint article details these findings and advocates for continued investigation into overcoming these limitations.

Accelerating Research with Simulation

Advancements in artificial intelligence are enabling social scientists to simulate human data, offering a potential solution to the inherent complexities of studying human behaviour. Traditional social science research, encompassing fields such as economics, psychology, sociology, and political science, is often constrained by the time-consuming, expensive, and difficult-to-replicate nature of working with human subjects. The application of large language models (LLMs) – AI systems capable of emulating human speech – allows researchers to inexpensively test assumptions, conduct preliminary studies, and estimate appropriate sample sizes.

These models offer the possibility of augmenting research pipelines by functioning as simulated participants, effectively increasing statistical power through a combination of human and LLM-generated data. LLMs are described as remarkably similar to people in their responses, allowing for their integration into various stages of the research process. While acknowledging the potential for bias and limited generalisability – detailed elsewhere – researchers, including Jacy Anthis, a visiting scholar at the Stanford Institute for Human-Centered AI, believe that even rudimentary applications of this technology are yielding promising results. Anthis’s work, presented in a recent preprint article, suggests that a focused year of further development could significantly enhance the utility of LLMs in social science research.