AI System Enables Scalable, Interactive Quantitative Surveys via Voice Technology

The challenge of gathering reliable quantitative data is being transformed with the advent of artificial intelligence. A team led by Danny D. Leybzon and Shreyas Tirumala of VKL Research, Inc., along with Nishant Jain and colleagues from SSRS, is pioneering a new approach using AI-powered telephone surveys. This research introduces an AI system capable of conducting full quantitative interviews, moving beyond simple automated responses to create a more natural and adaptive experience for respondents. Unlike traditional methods, this system leverages large language models and advanced speech technologies while adhering to rigorous research standards, such as randomised question order, to ensure data quality. The team’s validation studies, conducted with the SSRS Opinion Panel, demonstrate that this technology has the potential to improve survey completion rates, reduce drop-off, and enhance respondent satisfaction, offering a scalable and effective solution for quantitative data collection.

With the rise of voice-enabled artificial intelligence (AI) systems, quantitative survey researchers now have access to a new data-collection method: AI telephone surveying. By using AI to conduct phone interviews, researchers can scale quantitative studies while balancing the goals of human-like interactivity and methodological rigor. Unlike earlier automated systems, voice AI enables a more natural and adaptive respondent experience, proving robust to interruptions, corrections, and the nuances of human speech. The team built and tested an AI system to conduct quantitative surveys.

AI Adapts Surveys Using Conversational Interviews

This research explores how AI-powered conversational systems can effectively conduct telephone surveys, aiming to achieve results comparable to, or better than, traditional methods in terms of response rates, data quality, and respondent experience. Traditional telephone surveys are often expensive, time-consuming, and susceptible to interviewer bias, creating a need for scalable, cost-effective, and unbiased data collection methods. Researchers propose leveraging Large Language Models (LLMs) to create AI-powered conversational interviewers capable of conducting structured surveys, potentially revolutionizing survey methodology and offering a new approach to data collection. The study involved a pilot test with 104 adults from the SSRS Opinion Panel, with 70 participants completing at least one call.

Researchers conducted two waves of surveys, with the AI interviewer improved between the first and second waves based on initial results. Participants completed a real-world SSRS Opinion Panel Omnibus Survey, a 30-minute questionnaire with 123 questions, including sensitive topics such as personal finances, crime, and violence. Data collection involved automated telephone interviews conducted using the AI system, with measurements including completion rates, respondent satisfaction, and qualitative analysis of the AI-administered survey calls. Results demonstrate improvements in completion rates between the first and second waves, suggesting that technical improvements to the AI interviewer positively impacted performance.

An impressive 86% of respondents reported a neutral or positive experience with the AI interviewer, and a majority felt at least as comfortable speaking with the AI interviewer as with a human interviewer. Shorter surveys were associated with even greater respondent comfort. The AI interviewer excelled at handling ambiguous responses, maintaining robustness to respondent behaviors, and dealing with audio quality issues. However, challenges remain in achieving 100% accurate real-time transcription, balancing strictness and flexibility in accepting responses, and detecting respondent misbehavior, such as straightlining.

The study revealed that technical improvements between the waves led to increases in both completion and satisfaction metrics. Respondents did not appear less comfortable discussing sensitive topics with an AI interviewer compared to a human interviewer, and shorter surveys were associated with higher levels of respondent comfort. This research suggests that AI interviewers offer the potential for scalable, cost-effective survey data collection, reduced interviewer bias, and improved respondent experience. Future research should focus on larger-scale validation, investigating how different populations engage with AI interviewers, exploring inbound surveying scenarios, incorporating logic to detect respondent misbehavior, and assessing the effectiveness of AI interviewers in mixed-mode surveys.

AI System Completes Lengthy Telephone Survey

Recent advances in artificial intelligence have enabled the development of AI-powered systems capable of conducting quantitative telephone surveys, offering a new approach to large-scale data collection. Researchers successfully built and tested an AI interviewer, integrating automatic speech recognition, large language models, and speech synthesis to create a system that mimics human interaction during a survey. This system adhered to rigorous research standards, including question and answer randomization, ensuring methodological soundness comparable to traditional methods. The AI interviewer successfully conducted complex surveys with a nationally representative sample of U.

S. adults sourced from the SSRS Opinion Panel. Two waves of surveys were conducted, allowing for iterative improvements to the system based on initial results. Participants were informed they were speaking with an AI interviewer and given the option to discontinue the survey, prioritizing ethical considerations and transparency. This approach builds upon earlier automated voice systems, which often suffered from high break-off rates and discrepancies in responses compared to human interviewers, by leveraging the advanced capabilities of modern AI.

By employing large language models, the AI interviewer can interpret responses and navigate the complexities of a branching survey, responding to varied situations as a human interviewer would. Initial results suggest that shorter survey instruments and a responsive AI interviewer can contribute to improved completion rates, lower break-off rates, and increased respondent satisfaction. This represents a significant step towards scalable, cost-effective quantitative research, potentially unlocking new possibilities for gathering insights from large populations.

AI Surveys Maintain Rigor and Satisfaction

This research demonstrates the potential of artificial intelligence to conduct quantitative telephone surveys, offering a scalable method for data collection while maintaining methodological rigor. The AI interviewer successfully administered a representative survey of substantial length and complexity, with the majority of respondents reporting a neutral or positive experience. Improvements to the AI system between testing phases coincided with increased completion rates and respondent satisfaction, suggesting that ongoing technical development can further enhance performance. The study also indicates a relationship between survey length and respondent comfort, with those receiving shorter surveys expressing greater comfort with the AI interviewer.

However, the authors acknowledge limitations, including a small sample size for follow-up questions regarding comfort levels and the need for further investigation into how different demographic groups engage with AI interviewers. Future research should focus on larger-scale validation, exploring respondent preferences across various contexts, and assessing whether findings generalize to different sampling methods, such as random digit dialing. These steps will help determine the extent to which AI interviewers can contribute to more effective and representative survey research.