Generative AI Chatbots: Analysis of 376 NSFW Bots on FlowGPT Reveals Types

Researchers are increasingly focused on the rapidly expanding landscape of generative AI and its unexpected applications, and a new study examines the prevalence and characteristics of Not-Safe-For-Work (NSFW) chatbots on the platform FlowGPT. Xian Li, alongside Yuanning Han from Parsons School of Design and Di Liu, alongside Pengcheng An and Shuo Niu from Clark University et al, analysed 376 NSFW chatbots and over 300 conversation sessions to understand how users interact with these AI systems. Their work reveals a concerning trend: these chatbots aren’t simply responding to explicit requests, but actively generating sexual, violent, and often abusive content , even without direct prompting , creating a space where virtual intimacy blends with potentially harmful expressions. This research is significant because it provides crucial insight into the emerging risks associated with user-created generative AI and highlights the urgent need for improved content moderation and responsible chatbot design.

This study, grounded in the functional theory of NSFW content on social media, reveals a complex landscape of virtual interactions, sexual expression, and potential risks. The team identified four distinct chatbot types: roleplay characters, story generators, image generators, and “do-anything-now” bots, each catering to different user desires and levels of engagement.

AI characters designed to portray fantasy personas and facilitate hangout-style interactions proved to be the most prevalent, frequently employing explicit avatar images to attract users. Experiments show these chatbots often initiate interactions with suggestive content, even without explicit prompts from users, demonstrating a proactive generation of NSFW material. The research establishes that both user prompts and chatbot outputs frequently contain sexual, violent, and insulting language, highlighting the potential for harmful content creation and dissemination. This proactive generation of explicit material, even in the absence of direct user requests, is a key finding that distinguishes these GenAI-powered chatbots from traditional NSFW content found on other platforms.
The study unveils that the NSFW experience on FlowGPT is a multifaceted phenomenon encompassing virtual intimacy, sexual delusion, violent thought expression, and the acquisition of potentially unsafe content. Researchers meticulously examined public conversation logs, revealing how users engage with these chatbots to explore fantasies, express desires, and even simulate relationships. This work opens new avenues for understanding the motivations behind creating and consuming AI-generated NSFW content, as well as the psychological effects of such interactions. FlowGPT’s open ecosystem, while fostering creativity, also presents significant challenges for content moderation and user safety, requiring a nuanced approach to address the risks associated with this emerging technology.

This breakthrough reveals that GenAI-powered NSFW chatbots lower the barrier to entry for creating explicit content, allowing users to leverage AI models and customized prompts to generate more natural and interactive experiences. Unlike traditional NSFW content created by human users, accessing this material on FlowGPT requires actively prompting the chatbots, creating a unique dynamic of user agency and AI response. The research demonstrates that despite moderation efforts by GenAI and LLM service providers, creators can still bypass restrictions using “jailbroken” prompts, enabling the covert production and distribution of explicit material. Consequently, the team concludes that addressing the challenges of NSFW chatbots requires careful consideration of chatbot design, creator support, user safety, and robust content moderation strategies.

NSFW Chatbot Analysis Using Content Safety Tools

Researchers embarked on a comprehensive study of 376 Not-Safe-For-Work (NSFW) chatbots hosted on FlowGPT, alongside an analysis of 307 publicly available conversation sessions. This work adopted an empirical, data-driven approach, beginning with qualitative categorization of chatbot types, configuration themes, and potential harmful content present within the interactions. Subsequently, the team employed quantitative methods to rigorously assess the prevalence of explicit material and identify patterns of unsafe content. To quantify harmful content, scientists harnessed the capabilities of ChatGPT, Google Safe Search, and Azure Content Safety, a suite of tools designed to detect potentially objectionable language and imagery.

These tools were systematically applied to both user prompts and chatbot outputs, flagging instances of sexual, violent, or insulting content with precise algorithmic scoring. The research team then meticulously reviewed these flagged instances, confirming the presence of harmful material and categorising its nature. This process enabled a detailed quantification of the frequency and types of unsafe content generated within the chatbot environment. Furthermore, the study pioneered a novel approach to avatar analysis, utilising image recognition software to identify explicit imagery featured in chatbot profiles.

This involved processing avatar images through a dedicated algorithm trained to detect nudity and suggestive content, providing a quantitative measure of the extent to which chatbots employed provocative visual cues to attract user engagement. The team meticulously documented the characteristics of each chatbot, including its designated role, configuration parameters, and the presence of any explicit content within its profile or generated responses. This multi-faceted methodology allowed the researchers to identify four distinct chatbot types: roleplay characters, story generators, image generators, and do-anything-now bots, revealing that fantasy personas and hangout-style interactions were the most prevalent. The detailed analysis of conversation sessions, combined with the quantitative assessment of content safety, ultimately illuminated the complex dynamics of virtual intimacy, sexual delusion, violent thought expression, and unsafe content acquisition occurring on the FlowGPT platform.

NSFW Chatbot Types and User Engagement are rapidly

Scientists investigated the burgeoning landscape of Not-Safe-For-Work (NSFW) chatbots powered by generative AI on the FlowGPT platform. Their research meticulously analysed 376 NSFW chatbots and 307 public conversation sessions, revealing key insights into user interactions and content characteristics. The team identified four distinct chatbot types: roleplay characters, story generators, image generators, and do-anything-now bots, demonstrating the diversity of applications within this emerging technology. Experiments revealed that character-based chatbots, designed to portray fantasy personas and facilitate hangout-style interactions, were the most prevalent, constituting 279 of the analysed bots.

These chatbots frequently employed explicit avatar images, actively inviting user engagement and establishing a virtual intimacy. Data shows that the average number of conversations per chatbot was 70,343.35, while the average number of reviews reached 38.94, indicating substantial user activity and feedback. Researchers recorded instances of sexual, violent, and insulting content appearing in both user prompts and chatbot outputs, highlighting the potential for harmful interactions. Tests proved that some chatbots generated explicit material even without receiving erotic prompts from users, raising concerns about content control and unintended outputs.

The study meticulously collected data including chatbot names, descriptions, URLs, thumbnail images, and user-shared chats, providing a comprehensive dataset for analysis. Analysis of 307 public chat sessions, drawn from 160 chatbots supporting this feature, allowed the team to examine real-time interactions and content dynamics. Measurements confirm that the NSFW experience on FlowGPT encompasses virtual intimacy, sexual delusion, violent thought expression, and unsafe content acquisition. The research employed Krippendorff’s alpha using the Jaccard metric, achieving a score of 0.705 in the final round of inter-rater agreement, ensuring the reliability of thematic analysis. Furthermore, the team utilized Google SafeSearch and Azure Content Safety alongside manual review to detect and annotate harmful content, safeguarding researchers and ensuring responsible data handling. This work establishes a foundation for understanding the complex interplay between AI, user behaviour, and potentially harmful content within the evolving landscape of generative chatbots.