EU Funds Research to Make Machine Learning More Sustainable

Machine learning models are only as good as their latest training data, which can become outdated quickly. For instance, a model trained to recognize the current German Chancellor’s name would need to be retrained if there’s a change in leadership.

This retraining process is not only expensive but also demands enormous computational power, costing society more than just money. Dr. Sebastian Stich, a CISPA faculty member, aims to address this issue with his research project “Collective Minds,” which seeks to develop new algorithms for more sustainable and efficient machine learning. The European Research Council will fund his project for the next five years with approximately two million euros.

Stich’s approach focuses on collaborative learning, where small models share their knowledge rather than their training data, enabling them to adapt to new information and forget outdated knowledge more effectively. This could lead to a more sustainable, adaptable, and equitable form of machine learning that benefits not only large corporations but also smaller players, such as local hospitals in medicine.

Algorithms for Sustainable and Collaborative Machine Learning

Machine learning models are constantly faced with new data and changing tasks, requiring them to adapt and improve their abilities. However, this process typically involves retraining the models regularly, which is not only expensive but also demands enormous computational power, resulting in significant environmental costs. To address this issue, CISPA-Faculty Dr. Sebastian Stich has proposed a research project called “Collective Minds,” which aims to develop new algorithms for more sustainable and efficient machine learning.

The key to achieving this goal lies in better collaboration between models. According to Stich, the current approach of retraining models from scratch is extremely inefficient and unsustainable. This process is comparable to humans needing to relearn how to walk just to keep up with the latest political developments. The problem is further exacerbated by the phenomenon of “catastrophic forgetting,” where machine learning models tend to forget what they previously learned when confronted with new data or changing tasks.

Collaborative Learning: A New Approach

One way to address this issue is to move away from training one large and complex model towards training several smaller models that share their knowledge rather than their training data. This approach, known as federated learning, is a type of collaborative learning. Existing methods focus on using distributed training data to ultimately train a large model. However, Stich’s research project takes a different approach, aiming to enable small, independent models to collaborate effectively.

This new approach has several advantages. For instance, it can save resources and make machine learning more accessible to smaller players, such as local hospitals. Additionally, it can help protect sensitive data, conserve resources, and still allow models to improve collaboratively, ultimately providing better support in fields like cancer diagnostics.

Better Knowledge Sharing, Continuous Learning, and Meaningful Forgetting

The “Collective Minds” project has three core objectives. Firstly, the researchers aim to use improved training algorithms to make models better at adapting to new data and forgetting outdated knowledge. Secondly, they will work on enabling models trained on different devices and with various data sources to share their knowledge effectively. Each of these models develops a type of specialized expertise based on the data it was trained on and can be used for specific tasks.

Lastly, the researchers want to enable these small expert models to collaboratively tackle complex tasks. This would result in a more sustainable, adaptable, and equitable form of machine learning that benefits not only large corporations but also smaller players. The project’s outcomes could have significant implications for fields like medicine, where patient data is both scarce and sensitive.

EU Support for Innovative Research

The European Research Council (ERC) has awarded Dr. Sebastian Stich a Consolidator Grant to support his research project. This prestigious grant is an honor not only for Stich but also for his peers. The ERC supports outstanding researchers in establishing an independent research team, enabling them to implement their most promising research ideas.

Receiving an ERC Consolidator Grant is recognition of the researcher’s previous work and an incentive to continue developing innovative solutions to the challenges of machine learning. The EU’s support for innovative research is crucial in driving progress towards a more sustainable and equitable technological future.