Agent Data Protocol Unifies Diverse Datasets for Effective LLM Agent Fine-tuning, Overcoming a 20% Fragmentation Bottleneck

The increasing complexity of artificial intelligence agents demands effective training methods, yet progress remains hampered by fragmented and inconsistent data, a challenge Yueqi Song, Ketan Ramaneti, and Zaid Sheikh, from their respective institutions, address with a novel approach. They introduce the Agent Data Protocol (ADP), a unifying framework designed to bridge the gap between diverse agent training datasets, regardless of format or interface. This protocol streamlines the process of converting varied data into a standardized format, enabling more efficient and scalable training pipelines, and experiments demonstrate an average performance improvement of approximately 20% across multiple agent frameworks. By unifying thirteen existing datasets and achieving state-of-the-art results on key benchmarks without specialized tuning, the team, including Ziru Chen, Boyu Gou, and Tianbao Xie, significantly lowers the barrier to reproducible and effective agent training, paving the way for more capable and versatile AI systems.

Training Language Agents With Imbalanced Data

Researchers addressed a key challenge in training large language models (LLMs) to function as effective AI agents, focusing on the problem of imbalanced datasets. When training agents to perform tasks like coding, web browsing, and software engineering, datasets vary significantly in size, potentially causing models to prioritize information from larger sources. To overcome this, the team developed a technique to resample training data, using multipliers to increase or decrease the representation of specific datasets. This balanced approach ensures the model learns effectively from all available information, regardless of dataset size.

The researchers further refined this process by tailoring the training data to the specific capabilities of different agent frameworks. For example, agents designed for command-line and coding tasks received training exclusively on non-web data, while those focused on web interactions were trained only on web-based resources. Existing agent training datasets are fragmented and inconsistent, hindering their effective combination and utilization. ADP converts diverse datasets into a standardized format, simplifying the creation of large-scale, diverse training resources for various applications. Technically, ADP utilizes Pydantic schemas to define actions and observations relevant to common agent tasks, such as communication, browsing, coding, and tool use, coupled with automated validation to ensure data quality.

As a demonstration, the researchers converted thirteen existing agent datasets into the ADP format and created converters from ADP into three distinct agent architectures. This culminated in the creation and public release of the largest publicly available dataset for agent training, comprising 1. 3 million training trajectories, known as the ADP Dataset V1. Supervised fine-tuning using this standardized data resulted in an average performance gain of approximately 20% across diverse domains, achieving state-of-the-art or near state-of-the-art performance on standard benchmarks for coding, web browsing, research, and agentic tool use. Furthermore, the study revealed significant benefits from cross-task transfer learning, demonstrating that training on ADP data consistently improves performance compared to training on individual datasets.

Standardized Agent Data Improves Training Performance

Scientists achieved an average performance gain of approximately 20% over corresponding base models through large-scale supervised fine-tuning of AI agents, demonstrating a significant advancement in agent training methodologies. Researchers successfully converted a broad collection of thirteen existing agent training datasets into the ADP format, enabling seamless integration with multiple agent frameworks and facilitating the creation of a training-ready dataset. This work addresses a critical bottleneck in agent training, the lack of standardized data, by providing a common language for diverse data sources, including those focused on API/tool use, browsing, coding, software engineering, and general agentic workflows.

This standardization enabled the creation of the largest publicly available dataset for agent training, comprising 1. 3 million training trajectories, and allowed for consistent data processing without per-dataset engineering. Researchers successfully converted thirteen existing datasets into ADP format and demonstrated an average performance gain of approximately 20% when training agents using this standardized data. This approach achieved state-of-the-art or near state-of-the-art performance on standard coding, browsing, tool use, and research benchmarks, without requiring domain-specific adjustments. The key achievement lies in simplifying the process of integrating data from various sources into a single, usable format for agent training.

The team quantified the benefits of ADP, showing that converting data to the standardized format requires significantly less effort compared to working with datasets in their original, disparate forms. Looking ahead, researchers suggest extending ADP to incorporate multi-modal data, such as images and screen recordings, to capture more comprehensive agent-environment interactions. They also propose applying the standardization concept to evaluation settings to facilitate cleaner composition of datasets, agents, and evaluations.