Researchers at Columbia and the University of Cambridge have established a new benchmark for evaluating machine learning models predicting material properties, addressing concerns that some models arrive at correct answers “for the wrong reasons.” The benchmark focuses on accurately simulating a key factor in determining whether a model is truly physics-aware, according to Michele Simoncelli, assistant professor of applied physics at Columbia. Atoms, approximately one ten-billionth of a metre across, present immense computational challenges; however, this new standard is already being adopted by companies including Meta, Microsoft, Radical AI, and Orbital Materials. “We can call an atomistic ML model ‘physics-aware’ when it predicts the macroscopic properties of materials as a consequence of correctly describing the materials’ atomistic physics,” said Simoncelli.
Physics-Aware Benchmarking Evaluates AI Thermal Conductivity Predictions
The ability of machine learning models to accurately predict a material’s thermal conductivity hinges on simulating atomic vibrations, according to work from Columbia University and the University of Cambridge. Researchers assessed over 100 crystalline materials, discovering that some models accurately predicted both atomic vibrational properties and macroscopic thermal conductivity while others failed to do so, despite achieving similar accuracy in predicting formation energies and material stability. This discrepancy highlights a critical distinction between simply obtaining the correct answer and arriving at it through physically sound reasoning.
Balázs Póta, Paramvir Ahlawat, Gábor Csányi, and Michele Simoncelli’s team found that errors in force calculations, which determine how atoms respond to temperature or mechanical stress, can be masked when models focus solely on energy prediction. “But this can miss potential errors in forces, which determine the dynamics of atoms, which is particularly relevant to predict how they respond to temperature or mechanical perturbations,” Simoncelli explained.
The team’s benchmark specifically evaluates a model’s capacity to predict these vibrational characteristics, offering a more nuanced assessment than traditional methods. These models had been compared to each other mainly on their performance to predict energy, which is indeed the important quantity for most material properties.
The research underscores the importance of machine learning models that not only predict macroscopic properties but also accurately simulate the underlying physical processes. “Thermal conductivity is especially sensitive to the microscopic vibrational physics of a material, but the general idea is relevant to other physical properties as well, a topic which we are currently exploring in the group,” Simoncelli added.
Janosh Riebesell said, “I was very happy when Simoncelli’s group offered to contribute their test set and metric design to diversify the Matbench leaderboard.” The team’s work provides a signal for developers to optimize models, ensuring they arrive at solutions “for the right reasons,” and avoid potentially misleading predictions.
Thermal conductivity is especially sensitive to the microscopic vibrational physics of a material, but the general idea is relevant to other physical properties as well, a topic which we are currently exploring in the group.
Michele Simoncelli, assistant professor of applied physics at Columbia
Machine-Learning Interatomic Potentials Accelerate Quantum Calculations
Researchers recently assessed several machine-learning interatomic potentials, comparing their predictions against traditional quantum-mechanical calculations for over 100 crystalline materials to evaluate their ability to translate quantum characteristics into macroscopic physical properties. This assessment revealed discrepancies beyond simple predictive accuracy, highlighting that models can achieve correct macroscopic results despite flawed microscopic simulations. This finding underscores a critical distinction between arriving at the correct answer and doing so through a physically realistic simulation, a concern addressed by the new benchmark.
“There are cases in which ML models give apparently sensible predictions, but for the wrong reasons,” explained Póta, emphasizing the need for models grounded in fundamental physics. The research team further refined their models with material-specific training, achieving agreement within a few percent of reference calculations and, notably, agreement with experiments for lithium bromide.
This level of precision demonstrates the potential for machine learning to accelerate quantum calculations and improve the reliability of material property predictions. “Our benchmark provides a ‘physics-aware’ signal to ML developers to optimize their models,” said Póta, indicating a pathway toward more robust and trustworthy AI-driven materials science. The rapid adoption of this benchmark suggests a growing industry interest in ensuring models are not only accurate but also physically meaningful.
Our benchmark provides a ‘physics-aware’ signal to ML developers to optimize their models.
Póta, the graduate student who is first author of the article
Matbench Discovery Reveals Gaps in Modelled Atomic Vibrations
The newly established benchmark, incorporated into Matbench Discovery, quickly revealed performance discrepancies among machine learning models seemingly identical in their ability to predict crystal stability. Janosh Riebesell, creator and maintainer of Matbench Discovery, noted the tool exposed weaknesses in models lacking strong physical constraints, demonstrating that similar performance on one metric does not guarantee accuracy across all material properties. This interactive leaderboard now ranks models not only on stability and structure, but also on their prediction of thermal conductivity, providing a more comprehensive evaluation than previously available.
Beyond simply identifying inaccurate predictions, the benchmark highlights how models arrive at their answers, a distinction important for reliable materials discovery. Michele Simoncelli of Columbia University explained that a truly physics-aware model predicts macroscopic properties as a consequence of correctly describing atomic vibrations, a level of detail often missing in current approaches.
The rapid uptake of this benchmark signals growing industry interest in validating model accuracy beyond simple predictive power; Meta and Microsoft are already utilizing it to evaluate their machine learning models. With robotic labs increasingly closing the loop between prediction and synthesis, a model that misrepresents a material’s underlying physics risks discarding promising candidates and promoting unsuitable ones, underscoring the importance of this new evaluation standard.
We can call an atomistic ML model ‘physics-aware’ when it predicts the macroscopic properties of materials as a consequence of correctly describing the materials’ atomistic physics-namely, their atomic vibrations.
Michele Simoncelli, assistant professor of applied physics at Columbia
See today’s quantum computing news on Quantum Zeitgeist for the latest breakthroughs in qubits, hardware, algorithms, and industry deals.
