Twist2 System Enables 100% Successful Humanoid Data Collection in 15 Minutes with Portable, Mocap-Free Control

Humanoid robotics currently lags behind other areas of artificial intelligence due to a lack of comprehensive data collection methods, and researchers Yanjie Ze, Siheng Zhao from Amazon Frontier AI and Robotics, and Weizhuo Wang from Amazon Frontier AI and Robotics, alongside Angjoo Kanazawa, Rocky Duan from Amazon Frontier AI and Robotics, and Pieter Abbeel, address this challenge with the development of TWIST2. This new system provides a scalable and portable framework for collecting data that captures full-body human motions without the need for expensive motion capture equipment, representing a significant step forward in humanoid robotics research. TWIST2 utilises affordable virtual reality technology and a custom robotic neck to achieve holistic human-to-humanoid control, enabling the rapid collection of high-quality demonstrations, and the team successfully demonstrates complex skills such as dexterous manipulation and dynamic kicking, paving the way for more advanced and capable humanoid robots. The fully reproducible system and accompanying open-source dataset promise to accelerate progress in the field by providing researchers with the tools and data needed to develop increasingly sophisticated humanoid capabilities.

TWIST2 enables efficient data collection, rapid setup, and an enjoyable user experience compared to traditional motion capture solutions, while maintaining full whole-body control. The team developed a two-degree-of-freedom robotic neck, costing $250, to provide egocentric vision, allowing the robot to “see” as the human operator does. This combination empowers robots to perform long-horizon, dexterous, mobile whole-body manipulation and legged manipulation, achieving successful pick and place tasks in 15 to 20 minutes.
D Diffusion Policies for Humanoid Manipulation

This research details advancements in humanoid robot manipulation through improved 3D diffusion policies and a robust hardware/software framework. The core challenge lies in enabling humanoid robots to reliably perform complex manipulation tasks and generalize to new scenarios. Existing methods often struggle with real-world noise, variations in object properties, and the need for extensive training data. The researchers present a system combining improved 3D diffusion policies, which learn visuomotor policies to generate diverse and realistic actions based on visual input, and a comprehensive hardware/software framework.

The framework includes readily available and custom hardware, such as the PICO 4 Ultra VR headset for immersive visual feedback, PICO Motion Tracker for precise motion capture, Vive Trackers, and a custom-built robot platform. The software component, XRobotToolkit, is a cross-platform toolkit for teleoperation and control, alongside tools for motion tracking, data processing, and policy learning. A motion retargeting system, GMR, maps human demonstrations to the robot. Key innovations include enhanced 3D diffusion policies for better generalization and robustness, a flexible and user-friendly teleoperation framework, immersive visual feedback via the VR headset, precise motion capture with dedicated trackers, and motion retargeting for transferring human demonstrations.

The researchers also release several components as open-source resources, including the XRobotToolkit and GMR, to encourage further research. They created a dataset and benchmarking tools to facilitate the evaluation of different manipulation algorithms. This research represents a significant step towards more capable and versatile humanoid robots, accelerating progress in robotic manipulation and providing a promising path towards robots that can work alongside humans in real-world environments.

Real-time Human Control of Humanoid Robots

Scientists have developed TWIST2, a new system for controlling and collecting data from humanoid robots, achieving full whole-body control without expensive motion capture setups. The system tracks human movements in real-time using a PICO4U VR headset, translating them to the robot, and incorporates a custom-built, $250 two-degree-of-freedom robotic neck to provide egocentric vision. This combination enables holistic human-to-humanoid control, significantly advancing the scalability of humanoid robotics. Experiments demonstrate the system’s ability to collect 100 successful demonstrations of complex skills in just 15 minutes, with an almost 100% success rate.

Building on this data collection pipeline, researchers propose a hierarchical visuomotor policy framework that allows the humanoid robot to autonomously control its entire body based solely on egocentric vision. This policy successfully demonstrates both whole-body dexterous manipulation and dynamic kicking tasks, showcasing the potential for complex, autonomous behaviors. The team showcased the system’s capabilities through tasks such as consecutive whole-body pick and place operations and continuous kicking of a T-shaped box towards target regions, demonstrating the robot’s ability to perform long-horizon, complex movements. The entire system, including data and models, is open-sourced to ensure full reproducibility, allowing other researchers to build upon this work. Researchers achieved this by employing relative root translations and rotations, ensuring stable operation during extended tasks and precise control of lower-body movements.

Portable System Enables Humanoid Robot Teleoperation

The team presents TWIST2, a novel and portable system for collecting data and teleoperating humanoid robots without expensive motion capture technology. By integrating readily available virtual reality hardware with a custom-built robotic neck, researchers successfully capture full-body human motions and translate them to a humanoid robot, enabling complex manipulation and locomotion. Demonstrations show the system can reliably collect approximately 100 successful demonstrations in a short timeframe, and the resulting data supports the development of visuomotor policies capable of performing dexterous tasks, including manipulation and dynamic kicking. This work addresses a significant gap in humanoid robotics, where data collection is often hampered by the cost and complexity of existing systems.

The researchers highlight the importance of egocentric vision, a viewpoint mimicking human perception, and demonstrate that a low-cost neck attachment can provide roughly 80% of the functionality of more sophisticated setups. The team has made both the system design and the collected dataset openly available, fostering reproducibility and encouraging wider adoption within the research community. They acknowledge limitations related to the accuracy of motion tracking during highly dynamic movements and the inherent inaccuracies of the VR pose estimation. Future work will likely focus on addressing these limitations and expanding the capabilities of the visuomotor policies to include more flexible and adaptable behaviours.