Robot learning gains spatial awareness with equivariant transformer architecture.

EquAct, a novel transformer architecture, achieves state-of-the-art performance in 3D robotic manipulation by incorporating SE(3) equivariance. This ensures consistent behaviour under scene transformations, utilising a U-net with spherical Fourier features and SE(3)-invariant conditioning layers, demonstrated across simulation and real-world tasks.

Robotic manipulation frequently demands adaptability to variations in object pose and environment configuration. Researchers are now addressing limitations in current artificial intelligence systems by incorporating geometric principles directly into the learning process. A team led by Xupeng Zhu, Yu Qi, Yizhe Zhu, Robin Walters, and Robert Platt from Northeastern University present ‘EquAct: An SE(3)-Equivariant Multi-Task Transformer for Open-Loop Robotic Manipulation’. Their work details a novel transformer architecture designed to improve the spatial generalisation capabilities of robotic systems by enforcing equivariance to SE(3) transformations – meaning the system’s predictions change predictably when the scene is rotated or translated. This is achieved through a combination of spherical Fourier features and specifically designed layers that maintain consistency under these geometric changes, demonstrated through benchmarking on both simulated and real-world robotic tasks.

EquAct: Achieving Geometric Consistency in Robotic Manipulation

Transformer architectures have proven capable of learning conditioned, multi-task 3D open-loop manipulation policies from demonstrations, utilising both instructions and 3D observations. However, standard transformer networks lack inherent guarantees of geometric consistency, leading to unpredictable behaviour when the scene undergoes transformations in 3D space. These transformations are formally described by the Special Euclidean group SE(3), which encompasses both rotations and translations.

EquAct achieves SE(3) equivariance – meaning the system’s predictions change in a predictable way when the input undergoes a transformation – through two key architectural components. The system employs an efficient SE(3)-equivariant U-Net, which processes 3D point cloud data using spherical Fourier features. Spherical Fourier features provide a robust representation of 3D data, facilitating policy reasoning and maintaining geometric consistency. Furthermore, EquAct utilises SE(3)-invariant Feature-wise Linear Modulation (FiLM) layers to condition the policy. FiLM layers modulate the activations of the network based on the conditioning signal, and their invariance ensures that the instruction-derived signal remains consistent regardless of the scene’s orientation or position.

The performance of EquAct undergoes rigorous evaluation across both simulated and real-world scenarios. In simulation, the system achieves state-of-the-art results on 18 RLBench tasks, demonstrating its ability to generalise effectively under both SE(3) and SE(2) scene perturbations. SE(2) represents transformations in a 2D plane, encompassing rotations and translations. This showcases its potential for complex task execution. Researchers further validate the system’s capabilities through physical experiments, deploying it on four tasks: disassembling a pipe, plucking flowers at specified heights, picking fruits, and installing toilet rolls.

These results demonstrate that EquAct significantly improves spatial generalisation in robotic manipulation, offering a substantial improvement over existing methods. By explicitly incorporating SE(3) equivariance, the system exhibits predictable and reliable behaviour even when the scene undergoes transformations, enabling more robust and adaptable robotic systems. The design prioritises geometric understanding, allowing robots to operate effectively in dynamic and unpredictable environments, and paving the way for more sophisticated applications.