Bayesian optimization for scientists and engineers: From theory to application
In many scientific and engineering workflows, researchers must decide where to experiment since information and resources are often limited. Whether optimizing a chemical formulation, refining a manufacturing process, or improving product performance, researchers need to identify promising solutions with as few experiments as possible. Bayesian optimization has become one of the most effective methods for efficiently guiding experimentation towards the optimal regions of the design space. This guide explains what Bayesian optimization is, how it works, when to use it, and how it can be used across a wide range of industrial applications.
What is Bayesian optimization?
Bayesian optimization is an iterative learning method for optimizing a process or system. Instead of relying on a pre-planned experimentation strategy, it constructs a model from available data and uses that model to intelligently recommend where to experiment next to achieve the experimentation goals. Following each experiment, the model is updated based on the new results, improving its predictions and uncertainty estimates following each iteration. This learn-as-you-go approach allows Bayesian optimization to efficiently guide experimentation toward solutions that meet the desired goals while attempting to minimize the number of required experiments.
Why use Bayesian optimization?
Bayesian optimization is valuable when you want to learn sequentially in small batches, especially when experimentation is costly, time-consuming, or limited by how much starting data you have. It’s valuable in situations where you're working with a complex design space or only a handful of initial observations. Although it may not reach a "perfect" outcome, it can efficiently provide guidance toward the optimal test settings as it searches for regions of the design space that meet the specified goals of the experiment. Consider Bayesian optimization when:
- You have limited or messy starting data: The Bayesian Optimization platform in JMP Pro can begin with as few as two data points. It can also accommodate messy or partially missing historical data, including responses collected arbitrarily based on prior knowledge.
- Evaluation is expensive or slow: Bayesian optimization seeks to direct our experimentation toward settings that meet our product goals to reduce unnecessary testing.
- You have a complex response surface: Relationships within the data may be complex, involving high-dimensional inputs and potentially nonlinear relationships.
- You have a specific optimization target: Whether the goal is a maximum, minimum, or a target value, Bayesian optimization efficiently guides experimentation toward it.
- You want to test strategically, based on prior knowledge: Rather than testing broadly, you want to focus experiments in regions with the greatest promise for an optimal solution.
A simple analogy for how Bayesian optimization guides experimentation
Remember playing the “hot and cold” game as a kid? This simple game is a useful way to understand the core idea behind Bayesian optimization: how the model learns from each experiment to guide the next one.
Suppose you are tasked to find an object hidden in a dark room as quickly as possible. Someone tells you “warmer” when you move closer to the object and “colder” when you move further away. To find the object, you could search the room systematically, painstakingly checking every square foot. But a more efficient strategy is to adjust your next move based on whether each step was warmer or colder.
In other words, you learn from each attempt and use that information to guide the next step. Over time, this feedback leads you to the hidden object much faster than an exhaustive search.
In many industrial workflows, the same principle applies. Instead of finding a hidden object, the goal is often to optimize a mixture, formulation, or process for outcomes such as yield, strength, purity, or cost. Each experimental result provides updated information about the system. Bayesian optimization uses this feedback to guide the next experiment toward the objective, whether maximizing, minimizing, or targeting a specific value.
Bayesian optimization in action: A chemical formulation example
Suppose you have a chemical formulation and need to maximize its yield above 30 g/L. To be first to market, you need to find the ideal amounts of each chemical to maximize your yield efficiently, cutting out unnecessary experimentation. Similar to the dark room example above, you can use Bayesian optimization to learn from response data, which points you in the direction of the optimal formulation.
Assuming you have some historical data from past experiments, you use it to construct an initial model. In this example, there are three observations:
Next, the Bayesian Optimization platform in JMP Pro fits models to your data and recommends a new chemical formulation to maximize yield (Row 4). Use those three ingredient amounts to run the next experiment.
After recording the results in the data table (Row 4, Yield column), the platform learns from the most recent test results and suggests additional experiments to find an optimum.
Note that the Reason Added column identifies the platform’s strategy for selecting these test settings, in this case, Maximum Expected Improvement.
This guided process repeats until the product goals are satisfied.
In the final iteration (Row 9), the Reason Added column changes to Replicate Best Training Run, indicating that the optimum has likely been found. Notice the Yield has successfully been maximized above 30 g/L using only six iterations from the Bayesian Optimization platform.
JMP Pro’s Bayesian Optimization platform won’t spend too much time experimenting in undesirable regions that produce a low yield and where formulations don’t meet the goal. Instead, it seeks to find high-performing solutions that produce a maximum yield, while adequately searching the design space to ensure that the optimum has been found. The iterative process of testing and learning is a "smart" way to learn quickly, efficiently revealing an optimal formulation that meets all goals.
Can Bayesian optimization handle multiple responses?
The chemical formulation example above optimized a single response: yield. However, many real-world problems can't be reduced to a single measurable outcome. I've used Bayesian optimization for a single response, when tuning hyperparameters, but I've also encountered many projects where success depended on more than one outcome variable.
- Multi-response Bayesian optimization accounts for situations where more than one outcome must be optimized simultaneously.
- Multi-objective Bayesian optimization refers to situations where those responses have differing (and sometimes competing) objectives, such as maximizing yield while minimizing cost.
In many industries, both situations coexist. It’s not unusual to need to optimize several response variables at once, even when some of those variables have competing goals. Both multi-response and multi-objective Bayesian optimization extend the traditional framework to navigate these complex situations.
JMP Pro's Bayesian Optimization platform is designed specifically for these challenging scenarios of multiple responses and objectives, while providing the profilers, diagnostic tools, and interactive analysis features found throughout JMP.
How does Bayesian optimization work?
Bayesian optimization operates under two main concepts:
- A surrogate model that approximates the “unknown” function that needs to be estimated.
- Acquisition functions that tell Bayesian optimization where to sample next, given the current data and goals.
In this section, we will explore how Bayesian optimization works and introduce the machinery behind it.
Understanding the surrogate model
At the core of Bayesian optimization is the use of a surrogate model: a statistical model that attempts to approximate the unknown relationship between the factor and response variables. Most commonly, the unknown function is modeled by a Gaussian process (GP). The GP model provides both predictions and prediction uncertainty across the design space.
Gaussian process model for a single factor and response. The points in red are the observed data, black represents mean function, and gray shows regions of uncertainty.
A key feature of the Gaussian process model is that in areas with more data, there is less uncertainty. In areas without data nearby, there is a greater uncertainty of how the function behaves.
Leveraging acquisition functions
Since Bayesian optimization is an iterative experimentation approach, you need to know where to run your next test. To ensure you are sampling your design space with your goals in mind, Bayesian optimization uses acquisition functions.
Acquisition functions use the Gaussian process model to intelligently decide where to experiment next. Some acquisition functions prioritize regions that are likely to improve upon the best solution so far, while others find the most possible new information. More specifically, they use the Gaussian process to sample regions that show promise of containing the optimum or areas with high uncertainty (areas that have not been explored).
Gaussian process model with circled regions demonstrates a region of promise for finding a maximum and the areas that may need further exploration.
Putting it all together: The Bayesian optimization workflow
Bayesian optimization is an iterative process, meaning that it learns from past data and suggests the next best experiments to run. This flexibility allows this simple five-step workflow to be applied across industries and problems:
- Begin with initial data and goals.
- Use historical data or collect data using a starting design to build a GP surrogate model.
- Fit the Gaussian process model.
- Use acquisition functions to propose a new experiment(s) based on previous data/testing.
- Run the proposed experiments(s) and obtain responses.
- Repeat until goals are satisfied.
What are some examples of how Bayesian optimization is used?
Rather than testing every combination, Bayesian optimization constructs a model based on previous data to recommend promising next experiments. Bayesian optimization is widely used across industries where experimentation is expensive, time-consuming, or resource constrained.
By learning from previous results and guiding future experiments, Bayesian optimization helps teams improve processes, optimize formulations, and reduce unnecessary testing. Here are a few examples:
Chemical
- Identify temperature, solvent ratio, catalyst loading, and residence time combinations that maximize yield, without running every permutation.
- Balance multiple responses simultaneously (such as stability, purity, viscosity, and product performance) while managing tradeoffs between competing objectives.
- Reduce the number of lab experiments needed to land on a formulation that satisfies performance targets.
Semiconductor wafer fabrication
- Learn from previous experiments to identify etch time, chamber pressure, and temperature settings that maximize wafer yield within strict manufacturing constraints.
- Optimize multiple response variables such as yield, defects, power consumption, and performance to find settings that meet several goals at once.
- Prioritize the most informative process conditions for evaluation to minimize the number of test wafers needed to fine-tune process settings.
Pharmaceuticals
- Navigate mixture constraints, where ingredients must sum to 100 percent, to find formulations that satisfy dissolution and stability goals in fewer batches.
- Fine-tune mixing conditions, drying times, and temperature profiles while staying within the tight tolerances regulators expect.
- Trim the number of validation batches needed to lock in a consistent manufacturing process.
Consumer products
- Tune interacting variables such as temperature, pressure, cycle time, and feed rate, where each variable influences the optimal setting of the others.
- Concentrate testing on the regions of the design space most likely to pay off, instead of working through a grid of settings unrelated to the study's goals.
- Limit the number of full-scale production runs needed to validate a new process.
Biotechnology
- Find ideal feeding strategies, pH, and dissolved oxygen levels to improve cell growth.
- Balance competing objectives, such as maximizing protein yield while maintaining product quality, purity, and stability.
- Efficiently explore complex, nonlinear response surfaces where individual experiments can take weeks to evaluate.
Aerospace
- Fine-tune airfoil, propulsion, or composite material designs using fewer simulation runs, since a single CFD run can take hours or days on its own.
- Focus simulation effort on regions of the design space already performing well, refining designs efficiently rather than testing broadly.
- Support design validation by efficiently identifying a final design amid strict safety constraints and multiple engineering objectives.