Many engineering challenges lead to the same headache, which is having too many knobs to turn and too few opportunities to test them. Whether adjusting a power grid or designing a safer car, each evaluation can be expensive, and there may be hundreds of variables that could be important.
Consider the safety design of the vehicle. Engineers must integrate thousands of parts, and many design choices can affect how a vehicle performs in a crash. Classic optimization tools may start to struggle when searching for the best combination.
Researchers from MIT have developed a new approach that rethinks how the classic method, known as Bayesian optimization, is used to solve problems with hundreds of variables. In tests on realistic benchmarks of the engineering method, such as power system optimization, this approach found the best solutions 10 to 100 times faster than widely used methods.
Their technology leverages a basic model trained on tabular data that automatically identifies the most important variables to improve performance, and iterates the process to focus on better and better solutions. Baseline models are massive AI systems trained on vast public datasets. This allows them to adapt to different applications.
The researchers’ tabular basis model does not need to be constantly retrained because it works to find a solution, which increases the efficiency of the optimization process. This technology also provides greater speedup for more complex problems, so it can be particularly useful in demanding applications such as materials development or drug discovery.
“Modern AI and machine learning models can fundamentally change the way engineers and scientists create complex systems. We have come up with a single algorithm that can not only solve high-dimensional problems, but is also reusable so that it can be applied to many problems without having to start everything from scratch,” says Rosen Yu, a graduate student in computational science and engineering and lead author of the paper. Paper about this technique.
Yu is joined on the paper by Cyril Picard, a former MIT researcher and research scientist, and Faiz Ahmed, an associate professor of mechanical engineering and a core member of the MIT Center for Computational Science and Engineering. The research will be presented at the International Conference on Learning Representations.
Improving a proven method
When scientists seek to solve a multifaceted problem but have expensive methods to evaluate success, such as crash testing a car to see how good each design is, they often use a tried-and-true method called Bayesian optimization. This iterative method finds the best configuration of a complex system by building a surrogate model that helps estimate what to explore next while taking into account uncertainty in its predictions.
But the surrogate model must be retrained after each iteration, which can quickly become computationally intractable when the space of potential solutions is very large. In addition, scientists need to build a new model from scratch any time they want to address a different scenario.
To address both shortcomings, the MIT researchers used a generative AI system known as the tabular basis model as an alternative model within a Bayesian optimization algorithm.
“The tabular basis model is similar to ChatGPT for spreadsheets. The input and output of these models is tabular data, which is more commonly seen and used in engineering than language,” Yu says.
Just like large language models like ChatGPT, Claude, and Gemini, the model is pre-trained on a massive amount of tabular data. This makes it well equipped to address a range of forecasting problems. In addition, the model can be deployed as is, without any retraining required.
To make their system more accurate and efficient for optimization, the researchers used a trick that enables the model to determine which features of the design space will have the greatest impact on the solution.
“A car may have 300 design criteria, but not all of them are the main drivers of the best design if you are trying to increase some safety criteria,” Yu says. “Our algorithm can intelligently choose the most important features to focus on.”
This is done by using a tabular basis model to estimate which variables (or groups of variables) most influence the outcome.
The research then focuses on those variables with high impact rather than wasting time exploring everything equally. For example, if the size of the frontal crumple zone increases significantly and the vehicle’s safety rating improves, this feature will likely play a role in the improvement.
Bigger problems, better solutions
Yu says one of the biggest challenges they faced was finding the best tabular basis model for the task. They then had to connect it to a Bayesian optimization algorithm in such a way that it could identify the most salient features of the design.
“Finding the most salient dimension is a well-known problem in mathematics and computer science, but coming up with a method that takes advantage of the properties of the tabular basis model was a real challenge,” says Yu.
With the algorithmic framework in place, the researchers tested their method by comparing it to five state-of-the-art optimization algorithms.
On 60 benchmark problems, including real-world situations such as power grid design and car crash testing, their method consistently found the best solutions 10 to 100 times faster than other algorithms.
“When the optimization problem gets more and more dimensional, our algorithm really shines,” Yu added.
But their method does not outperform the baselines for all problems, such as automated path planning. This likely indicates that the scenario was not well specified in the model’s training data, Yu says.
In the future, researchers would like to study methods that can enhance the performance of tabular basis models. They also want to apply their techniques to problems with thousands or even millions of dimensions, such as designing a marine ship.
“At a higher level, this work points to a broader shift: the use of fundamental models not just for perception or language, but as algorithmic engines within scientific and engineering tools, allowing classical methods like Bayesian optimization to expand into systems that were previously impractical,” Ahmed says.
“The approach presented in this work, using a pre-trained baseline model with high-dimensional Bayesian optimization, is an innovative and promising way to reduce the heavy data requirements of simulation-based design,” says Wei Chen, the Wilson-Cook Professor of Engineering Design and chair of the Department of Mechanical Engineering at Northwestern University, who was not involved in this research. “Overall, this work is a practical and powerful step toward making advanced design optimization more accessible and easier to apply in real-world settings.”
(Tags for translation)Rosen Yu
