Multi-Robot Lunar Tests: Framework Achieves KPI for Three Realistic Scenarios

Researchers are tackling the challenge of consistently evaluating multi-robot field tests designed to prepare for lunar and planetary resource prospecting. Julia Richter, David Oberacker from FZI Research Center for Information Technology, and Gabriela Ligeza from the University of Basel, alongside Valentin T. Bickel et al., present a new framework of Key Performance Indicators (KPIs) to address the current lack of standardised evaluation methods. This work is significant because it moves beyond scenario-specific engineering metrics, instead linking field performance directly to overarching scientific objectives, such as identifying resources like ilmenite and water ice. By prioritising efficiency, robustness and precision across realistic multi-robot scenarios, the team offers a practical tool for comparing trials and accelerating the development of robotic systems for future planetary exploration.

This research addresses a critical gap in the field, where comparing results across different robotic platforms and experimental setups has proven challenging. The team established a structured set of metrics derived from three realistic, multi-robot lunar scenarios, ilmenite prospecting in volcanic plains, rare-earth-element exploration of ejecta blankets, and the search for polar water ice. This innovative approach prioritises scenario-dependent factors like efficiency, robustness, and precision, ensuring practical applicability for future field deployments.

The study directly links robotic performance to overarching science-driven objectives, moving beyond traditional engineering metrics that often lack clear scientific relevance. Researchers co-developed these scenarios with both lunar geologists and roboticists, guaranteeing that the KPIs accurately reflect both operational feasibility and scientific value. Validation of the framework occurred during a multi-robot field test, demonstrating its practicality and ease of use for assessing efficiency and robustness. While KPIs focused on precision require reliable ground-truth data, often difficult to obtain in outdoor analog environments, the framework proved effective in evaluating core performance aspects.
This work establishes a common evaluation standard, enabling consistent and goal-oriented comparisons of multi-robot field trials. By systematically analysing robotic systems in simulated lunar environments, scientists can identify weaknesses and opportunities for improvement in the development of future planetary exploration technologies. The framework considers the diverse lunar terrain, ranging from smooth plains to steep crater walls, and accounts for the unique challenges posed by harsh environmental conditions. Heterogeneous robotic teams, combining different locomotion and sensing capabilities, are central to this approach, offering redundancy, increased science acquisition rates, and enhanced scientific depth.

Experiments show that multi-robot systems offer benefits such as improved performance in extreme terrain, which is often inaccessible to single rovers. The research builds upon recent advancements in legged platforms and the growing interest in cooperative robotics, as evidenced by competitions like the DARPA SubT and the ESA, ESRIC Space Resources Challenge. The team’s framework moves beyond simply assessing robotic mobility and mapping capabilities, instead focusing on the ability to achieve specific scientific goals within realistic lunar prospecting missions. The research team derived these KPIs from three realistic, multi-robot scenarios mirroring scientific objectives and operational constraints encountered on the Moon, specifically focusing on ilmenite, rare earth elements, and water ice exploration. These scenarios were meticulously designed to reflect the diverse terrain and harsh environmental conditions present on the lunar surface, including smooth mare plains and steep crater walls. Experiments employed a combination of wheeled and legged platforms, acknowledging the limitations of single-platform approaches in extreme lunar terrain such as unconsolidated regolith and steep slopes. The team assessed performance across the defined KPIs, utilising data gathered during simulated lunar prospecting tasks, and found the framework readily applicable for evaluating efficiency and robustness. However, the study revealed that precision-oriented KPIs required reliable ground-truth data, which proved challenging to obtain consistently in outdoor analog environments. To facilitate this, the team harnessed orbital data, including LROC low-incidence image mosaics, elevation profiles, and maps showcasing thermophysical, topographic, and compositional properties, to establish preliminary prospecting grids and assess performance against known lunar features. The research team derived these KPIs from three realistic, multi-robot scenarios mirroring scientific objectives and operational constraints on the Moon, specifically targeting ilmenite, rare earth elements, and water ice. This work directly responds to the increasing interest in sustainable lunar presence and the need for robust robotic exploration methods in challenging terrain. Experiments focused on defining metrics related to efficiency, robustness, and precision, prioritised according to the specific demands of each scenario.

The framework was validated through a multi-robot field test, demonstrating its practicality and ease of application for efficiency and robustness-related KPIs. Measurements confirmed the framework’s utility in assessing how effectively robots complete tasks and withstand environmental challenges during simulated lunar missions. However, the study revealed that precision-oriented KPIs require reliable ground-truth data, which proved difficult to obtain consistently in outdoor analog environments. Results demonstrate that the proposed framework facilitates consistent, goal-oriented comparison of multi-robot field trials, enabling systematic development of robotic systems for future planetary exploration.

The team successfully formulated KPIs based on scientifically grounded operational scenarios, co-developed by lunar geologists and roboticists, ensuring both scientific relevance and operational measurability. This approach reverses the conventional method of deriving KPIs from engineering trials, offering a more targeted and effective evaluation process. The research highlights the benefits of heterogeneous robotic teams, combining different locomotion and sensing capabilities to improve redundancy, science acquisition rates, and scientific depth. Tests showed that multi-robot systems offer advantages over single-platform approaches, particularly in extreme lunar terrains such as unconsolidated regolith and steep slopes. The framework’s emphasis on scenario-dependent priorities allows for tailored evaluation, recognising that the optimal balance between efficiency, robustness, and precision varies depending on the specific mission objectives and environmental conditions. This breakthrough delivers a common evaluation standard for robotic prospecting missions.

Lunar robot KPI framework validated in trials

Scientists have developed a structured framework of key performance indicators (KPIs) to evaluate multi-robot teams designed for lunar resource prospecting. This research addresses the need for standardised evaluation methods in field trials, which currently suffer from inconsistencies due to varying robot platforms and experimental setups. The framework prioritises efficiency, robustness, and precision, tailoring these priorities to specific mission scenarios targeting ilmenite, rare earth elements, and water ice. The proposed KPIs were validated in a real-world multi-robot field test, demonstrating practicality for assessing efficiency and robustness.

However, the authors acknowledge that evaluating precision-oriented KPIs requires reliable ground-truth data, which can be challenging to obtain in outdoor analog environments. The framework allows for meaningful comparisons between different robotic systems and supports informed mission design by acknowledging trade-offs between performance aspects like exploration speed and data accuracy. This work establishes a common evaluation standard for robotic lunar prospecting, enabling consistent assessment of multi-robot field trials and facilitating systematic development of robotic systems for future planetary exploration. While most KPIs measure overall mission performance rather than specific inter-robot coordination, they are designed to be applicable to diverse, multi-robot teams through aggregation and shared objectives. The authors suggest future work could focus on improving the acquisition of ground-truth data in analog environments to enhance the evaluation of precision-related KPIs.