MIT researchers have significantly accelerated design optimization across complex engineering challenges, achieving solutions 10 to 100 times faster than conventional methods in initial tests on benchmarks like power-system optimization. The team re-engineered Bayesian optimization, a widely used technique, by integrating a foundation model trained on tabular data to automatically pinpoint the most critical variables influencing performance. This allows the algorithm to efficiently refine solutions without constant retraining, a benefit particularly valuable for demanding fields such as materials development and drug discovery. “A car might have 300 design criteria, but not all of them are the main driver of the best design if you are trying to increase some safety parameters,” explains Rosen Yu, a graduate student in computational science and engineering and lead author of the research, which will be presented at the International Conference on Learning Representations.
Bayesian Optimization Enhanced with Tabular Foundation Models
The innovation addresses a critical bottleneck in design processes where evaluating potential solutions, such as crash-testing a vehicle, is resource intensive and time-consuming. The core of the advance lies in replacing the traditionally retrained surrogate model within Bayesian optimization with a generative AI system specifically designed for tabular data. Unlike models requiring constant retraining, this foundation model, pre-trained on vast datasets of structured information, adapts readily to new applications. This reusability is a key benefit, allowing engineers to apply the algorithm to diverse problems without starting from scratch. Crucially, the algorithm doesn’t simply accelerate existing methods; it intelligently prioritizes which variables to explore. By identifying high-impact variables, like the size of a front crumple zone and its effect on safety ratings, the system avoids wasting computational effort on less influential parameters.
While the method did not outperform baseline algorithms in all scenarios, such as robotic path planning, researchers believe this is due to limitations in the model’s training data. Faez Ahmed, associate professor of mechanical engineering, notes that this work “points to a broader shift: using foundation models not just for perception or language, but as algorithmic engines inside scientific and engineering tools.”
Foundation Model Identifies Key Design Variables
Engineers confronting complex design challenges routinely face a daunting number of variables, hindering efficient optimization. Traditional methods struggle when evaluating countless combinations of parameters, particularly in fields like power grid management and vehicle safety. While Bayesian optimization offers a structured approach to navigate these complexities, its reliance on repeatedly retraining surrogate models presents a significant computational bottleneck, especially as the number of variables increases. Researchers at MIT have addressed this limitation by integrating a novel application of foundation models, large AI systems pre-trained on extensive datasets, into the Bayesian optimization framework. This innovation centers on a “tabular foundation model,” described by Rosen Yu as “like a ChatGPT for spreadsheets,” capable of processing and predicting outcomes based on structured, tabular data common in engineering applications. Unlike conventional surrogate models requiring constant recalibration, this pre-trained model offers reusability, substantially accelerating the optimization process. The team’s approach doesn’t simply apply the model; it leverages its capacity to identify the most influential design variables.
“Modern AI and machine-learning models can fundamentally change the way engineers and scientists create complex systems. We came up with one algorithm that can not only solve high-dimensional problems, but is also reusable so it can be applied to many problems without the need to start everything from scratch,”
Rosen Yu, a graduate student in computational science and engineering and lead author of a paper on this technique
100x Speedup Achieved on Engineering Benchmarks
The team, led by computational science and engineering graduate student Rosen Yu, has integrated a pre-trained tabular foundation model into a Bayesian optimization algorithm, dramatically accelerating the process of finding optimal solutions for multifaceted problems. This advancement addresses a longstanding challenge; evaluating every possible configuration in areas like power grid optimization or vehicle design is often prohibitively expensive and time-consuming. Unlike traditional surrogate models that require constant retraining, this foundation model leverages its pre-existing knowledge base, eliminating a significant computational burden. This reusability is particularly valuable when shifting between different design scenarios, as the model doesn’t need to be rebuilt from scratch each time. The team demonstrated the efficacy of their approach on 60 benchmark problems, including simulations of power grid design and car crash testing, consistently outperforming five state-of-the-art optimization algorithms. A key element of the system’s efficiency is its ability to intelligently prioritize design variables.
By identifying the variables with the greatest impact on performance, the algorithm avoids wasting computational resources on less influential parameters. While the method did not excel in all tested scenarios, robotic path planning proved a current limitation, the researchers are already exploring ways to further enhance the performance of tabular foundation models and scale the technique to even higher-dimensional problems, potentially reaching applications like naval ship design.
“The approach presented in this work, using a pretrained foundation model together with high‑dimensional Bayesian optimization, is a creative and promising way to reduce the heavy data requirements of simulation‑based design. Overall, this work is a practical and powerful step toward making advanced design optimization more accessible and easier to apply in real-world settings,”
Wei Chen, the Wilson-Cook Professor in Engineering Design and chair of the Department of Mechanical Engineering at Northwestern University
Reusable Algorithm Scales High-Dimensional Optimization Problems
The ability to rapidly optimize complex designs is becoming increasingly vital across industries, and a team at MIT has developed an algorithm poised to accelerate progress in fields ranging from power grid management to vehicle engineering. Researchers tackled this challenge by reimagining Bayesian optimization, a method for finding the best configuration for a complicated system, and integrating it with a novel application of artificial intelligence. This pre-training allows the model to adapt to different applications without needing to be rebuilt from scratch for each new scenario. A key advancement lies in the algorithm’s ability to intelligently prioritize variables. Recognizing that not all design criteria are equally influential, the system identifies the most critical features impacting performance.
“A car might have 300 design criteria, but not all of them are the main driver of the best design if you are trying to increase some safety parameters. Our algorithm can smartly select the most critical features to focus on,”
