Loihi 2 Neuromorphic Hardware Enables Autonomous Robot Control with Reinforcement Learning and Spiking Neural Networks

Researchers are increasingly focused on creating robotic systems capable of autonomous decision-making, and a new study demonstrates a significant step towards achieving this goal with neuromorphic computing. Kenneth Stewart, Roxana Leontie, and Samantha Chapin, alongside colleagues at the U. S. Naval Research Laboratory and Intel Labs, present a complete system for translating artificial intelligence algorithms into a form suitable for deployment on Intel’s Loihi 2 neuromorphic hardware. The team successfully converted a reinforcement learning policy, originally trained to control the Astrobee free-flying robot, into a spiking neural network and ran it on the Loihi 2 chip, achieving low-latency and energy-efficient performance in a realistic simulation. This work establishes a pathway towards creating truly autonomous robots capable of operating in complex environments with minimal power consumption, paving the way for future applications in both space exploration and terrestrial robotics.

Sim-to-real transfer for space robotics

This research focuses on developing intelligent robots for use in space, employing artificial intelligence, specifically reinforcement learning and spiking neural networks, to enable autonomous operation. A key challenge addressed is transferring learned behaviors from simulated environments to the complexities of real-world space conditions, such as unpredictable movements and sensor inaccuracies. The work also prioritizes energy efficiency, crucial for space applications where power is limited, and explores spiking neural networks as a potential solution due to their event-driven processing. Researchers utilize reinforcement learning to train robots to perform tasks through trial and error, employing a technique called Proximal Policy Optimization.

Spiking neural networks, inspired by the human brain, communicate using discrete events, potentially offering lower energy consumption than traditional deep learning approaches. To bridge the gap between simulations and reality, the team converts continuous-valued networks into efficient, event-driven spiking networks, leveraging sophisticated simulation environments, including NVIDIA’s Isaac Sim and a unified framework called Orbit, to train and test robotic systems. The research explores controlling robotic arms and free-flying spacecraft using these AI techniques, focusing on tasks like satellite servicing, on-orbit assembly, and robotic manipulation. The team aims to demonstrate that these systems can significantly reduce energy usage compared to traditional AI approaches, paving the way for more capable and efficient space robots.

Results indicate that spiking neural networks hold promise as a low-power alternative for robotic control, although bridging the simulation-to-real gap remains a significant challenge. Neuromorphic chips, like Intel’s Loihi 2, can accelerate spiking neural network computations and further reduce energy consumption. This work contributes to the advancement of AI for robotics and enables the development of more efficient robots for space exploration and utilization, with future work focusing on robust transfer techniques and scaling up AI systems for complex tasks.

Spiking Networks Control Robot Policy in Simulation

This study presents a complete pipeline for deploying reinforcement learning policies onto neuromorphic hardware via conversion into spiking Sigma-Delta Neural Networks. To rigorously evaluate performance, the converted policy was deployed within NVIDIA’s Omniverse Isaac Lab simulation environment, allowing for closed-loop control of the robot’s motion and direct comparison with traditional GPU-based execution. The core of the methodology involves training a neural network using Proximal Policy Optimization, employing techniques to optimize the learning process.

This network was trained to guide the Astrobee to desired positions and orientations in zero-gravity, utilizing a reward function that incentivizes accurate movement. Crucially, the team re-trained the policy using Rectified Linear Units to ensure compatibility with the subsequent conversion to a spiking neural network. The conversion process begins by transforming the input layer into a Delta layer, transmitting only information exceeding a reference value. Hidden layers are transformed into Sigma-Delta-ReLU layers, and the output layer becomes a Sigma layer, effectively translating continuous activations into sparse, event-driven spikes. This conversion leverages a thresholding mechanism to ensure efficient information transmission within the neuromorphic system. The team then deployed and evaluated the converted network on the Loihi 2 processor within the Isaac Lab simulation, demonstrating the feasibility of utilizing neuromorphic hardware for robotic control applications.

Neuromorphic Control of Robots via Reinforcement Learning

Scientists have successfully demonstrated a pipeline for deploying reinforcement learning-trained artificial neural networks onto neuromorphic hardware by converting them into spiking Sigma-Delta Neural Networks. This work establishes a pathway toward low-latency and energy-efficient processing, validated through simulated control of an Astrobee free-flying robot within the NVIDIA Omniverse Isaac Lab environment. The team converted a reinforcement learning policy, initially trained with Rectified Linear Units, into a spiking neural network and deployed it on Intel’s Loihi 2 architecture for closed-loop control of the robot’s motion. Experiments reveal the feasibility of utilizing neuromorphic platforms for robotic control, capitalizing on the ease of training artificial neural networks and their potential for transferring learned behaviors to real-world scenarios while exploiting the energy efficiency of spiking neural networks.

This research addresses critical power constraints in robotic applications, particularly in space and mobile environments where energy demands can hinder deployment. The team’s approach leverages Loihi 2’s support for quantized graded spikes, enabling conversion of a trained artificial neural network into a quantized graded-spike spiking neural network that utilizes both temporal and spatial sparsity. This work builds upon prior research demonstrating hardware validation of a similar controller in space, extending its capabilities through the implementation of a low-power neuromorphic solution. The successful conversion and deployment of the spiking neural network on Loihi 2 establishes a foundation for future advancements in energy-efficient, real-time computation for both space and terrestrial robotics. By demonstrating this pipeline, scientists pave the way for wider adoption of neuromorphic computing in robotics, addressing resource constraints and enabling long-term, autonomous operation in demanding environments.

Neuromorphic Control of Free-Flying Robots Demonstrated

This research demonstrates the successful deployment of a reinforcement learning policy, originally trained for controlling a free-flying robot, onto neuromorphic hardware using spiking Sigma-Delta Neural Networks. The team converted an artificial neural network policy, previously validated in simulations and on the International Space Station, into a format compatible with Intel’s Loihi 2 architecture. Evaluation within a simulated environment confirmed the feasibility of executing the control policy on the neuromorphic hardware, establishing a pathway towards energy-efficient and low-latency robotic control. The results highlight the potential of this approach for future space and terrestrial robotics applications, where energy consumption and real-time performance are critical.

While some performance deviation was observed compared to traditional GPU execution, the researchers note this may be acceptable when considering the benefits in energy efficiency and latency. The team acknowledges that larger and more complex control policies will likely exhibit different resource requirements, potentially amplifying the advantages of neuromorphic execution. Future work will focus on mitigating the observed performance limitations and further optimizing the system for more demanding applications. This research demonstrates a promising pathway towards developing more efficient and capable robots for a wide range of applications, particularly in environments where energy and computational resources are limited.