Agentif-Oneday Achieves 80.1% Success in Daily Tasks for General AI Agents

Researchers are tackling the challenge of building AI agents truly useful for everyday life, despite recent advances in complex problem-solving. Kaiyuan Chen (xbench.org), Qimin Wu, and Taiyu Hou (xbench.org), et al., introduce AgentIF-OneDay, a new benchmark designed to rigorously test an agent’s ability to follow natural language instructions and complete diverse, real-world tasks , from managing workflows to interpreting attachments and refining ongoing projects. This work is significant because current AI evaluations often focus on difficulty, overlooking the breadth of skills needed for daily assistance, and AgentIF-OneDay provides a comprehensive assessment across 104 tasks and 767 scoring points, revealing that both API-driven and reinforcement learning-based agents are currently leading the field. The benchmark’s robust evaluation pipeline, combining LLM verification with human judgement, demonstrates an impressive 80.1% agreement rate, paving the way for more practical and user-centric AI agent development.

AI agent evaluation via daily life tasks

Scientists have unveiled AgentIF-OneDay, a novel benchmark designed to rigorously evaluate the capacity of AI agents to handle the diverse and nuanced demands of everyday life. This groundbreaking work addresses a critical gap in current AI evaluation methods, which often prioritise increasing task complexity without adequately reflecting the breadth of activities encountered in daily work, learning, and personal life. The research team proposes a comprehensive framework to determine whether general users can effectively leverage natural language instructions and AI agents to complete a wide array of practical tasks, extending beyond simple problem-solving to encompass understanding various file attachments and delivering tangible, file-based results. AgentIF-OneDay is structured around three user-centric categories, Open Workflow Execution, Latent Instruction, and Iterative Refinement, each designed to assess distinct facets of agentic intelligence.
The study meticulously assesses an agent’s ability to adhere to explicit and complex workflows in the Open Workflow Execution category, testing its robustness in processing long-context instructions and avoiding common pitfalls like instruction forgetting and hallucination. Furthermore, the Latent Instruction dimension challenges agents to infer implicit rules and constraints from provided attachments, demanding a higher level of reasoning and contextual understanding. Crucially, the Iterative Refinement category simulates a collaborative human-agent interaction, requiring the agent to precisely modify and expand upon existing work based on user feedback, thereby evaluating its state maintenance and collaborative capabilities. The benchmark comprises 104 tasks, encompassing a total of 767 scoring points, ensuring a thorough and granular assessment of agent performance.

Researchers employed instance-level rubrics and a refined evaluation pipeline, aligning LLM-based verification with human judgment to ensure accuracy and reliability, achieving an impressive 80.1% agreement rate using Gemini-3-Pro. This innovative approach incorporates visual parsing for file types like PPT and HTML, leveraging Vision-Language Models (VLMs) as verifiers and utilising a Search Mode for real-time verification requirements. Benchmarking four leading general AI agents, the team discovered that agent products built on APIs and ChatGPT agents utilising agent RL consistently achieve top-tier performance, demonstrating the potential of both approaches. This work establishes that leading LLM APIs and open-source models have demonstrably internalised agentic capabilities, empowering application teams to develop cutting-edge AI agent products. The high-quality, finely annotated instruction data generated by AgentIF-OneDay also holds significant promise as valuable training data for reinforcement learning, potentially accelerating further advancements in the field. Ultimately, this research not only reveals the strengths and limitations of current AI agents but also provides a crucial resource for optimising future agent development and bridging the gap between theoretical capabilities and practical, everyday utility.

AgentIF-OneDay benchmark creation via automated pipeline

Scientists developed AgentIF-OneDay, a novel benchmark comprising 104 tasks and 767 scoring points, to rigorously evaluate the capacity of AI agents to handle diverse, daily tasks. The research team engineered a File-centered Automated Agentic Pipeline for task generation, beginning with seed tasks manually annotated by experts. Utilizing the ChatGPT agent, they collected information-dense attachments with high potential for question generation, effectively expanding the scope of potential evaluation items. Subsequently, workflow frameworks were extracted and augmented from these human-annotated seeds, providing a structured foundation for task creation.

The study pioneered a synthesis process combining attachments and workflows, resulting in logically coherent evaluation items deeply correlated with the attachment information. This methodology isn’t merely evaluative; it’s extensible to broader agentic data synthesis applications, offering a versatile tool for future research. To ensure robust and consistent evaluation, the team employed instance-level rubrics and a refined evaluation pipeline leveraging LLM-based verification alongside human judgment. Specifically, they achieved an 80.1% agreement rate between LLM and human assessments using0.6% consistency with human scoring when employing Gemini-3-Pro.

Benchmarking four leading general agents revealed that both API-based products and ChatGPT agents built using agent RL currently represent the leading tier of performance. This work highlights that leading LLM APIs and open-source models have internalized agentic capabilities, empowering application teams to develop cutting-edge Agent products. The high-quality, finely annotated instruction data generated by AgentIF-OneDay holds significant potential as training data for reinforcement learning, promising further advancements in AI agent development.

AgentIF-OneDay benchmark evaluates everyday AI agent performance

Scientists have unveiled AgentIF-OneDay, a new benchmark comprising 104 tasks and 767 scoring points designed to rigorously evaluate AI agents in everyday scenarios. The research demonstrates a significant step towards bridging the gap between advanced agent capabilities and practical user experience, moving beyond evaluations focused solely on increasing task difficulty. Experiments revealed that the framework effectively assesses an agent’s ability to handle diverse daily activities, requiring both dialogue-based problem-solving and the delivery of tangible, file-based outputs. The team measured agent performance across three key categories: Open Workflow Execution, Latent Instruction Inference, and Iterative Refinement.

In Open Workflow Execution, agents were tested on their adherence to explicit, complex workflows, with the study focusing on long-context processing and minimising instances of instruction forgetting or hallucination. Latent Instruction Inference challenged agents to autonomously deduce implicit rules from task attachments and apply them to new tasks, demonstrating their capacity for nuanced understanding. Results demonstrate that agents must also excel in Iterative Refinement, a collaborative process where they modify existing work based on user feedback, assessing state maintenance and human-machine synergy. Measurements confirm an 80.1% agreement rate between LLM-based verification using Gemini-3-Pro and human judgment, highlighting the reliability of the evaluation pipeline.

This was achieved through instance-level rubrics and a refined pipeline incorporating visual parsing for PPT and HTML files, alongside Vision-Language Models as verifiers. Tests prove the efficacy of a File-centered Automated Agentic Pipeline for task generation, utilising ChatGPT to collect information-dense attachments and augment workflow frameworks. The breakthrough delivers a robust methodology for assessing agent capabilities, revealing that both agent products built on APIs and ChatGPT agents leveraging agent RL currently represent the leading tier of performance. Data shows that leading LLM APIs and open-source models have demonstrably internalised agentic capabilities, empowering application teams to develop cutting-edge AI agent products. This work provides a valuable resource for optimising AI agents and offers a high-quality, finely annotated dataset for reinforcement learning applications.