In conversations with users new to design of experiments (DOE), we occasionally hear that they were unable to solve their problems: “My DOE failed, I was not able to fit a model,” or “None of my factors were significant,” or “My DOE didn’t work because I had missing runs, so I switched back to one-factor-at-a-time to get results.”

There are many reasons DOEs can “fail,” so it is helpful to focus instead on what was learned. These lessons are best learned once you reframe DOE as an incremental process, one that is a step-by-step approach instead of one-shot magic. Let’s explore some common challenges that might occur when using DOE and how they can be used to explore the process of interest and raise important questions to ask. This first installment in the three-part blog series is about missing runs, how they influence the design’s ability to fit a good model and possible ways to act after they occurred.

The curse of missing runs

When runs go missing: A real-world scenario

The R&D engineers of a chemical company want to optimize a newly developed process with three factors for yield and have already finished their DOE. While experimenting, they noticed that some of the runs can’t be measured because of unexpected decomposition of their chemical solution or no reaction at all due to insufficient process times and temperatures. The runs were treated as missing values in the design since they could not be measured. What does this mean?

How missing runs corrupt your model

The engineers fear that the missing runs will have negative effects on their model and that the gathered data will be useless. It is a very reasonable concern, since DOEs are constructed with every run to be as impactful as possible for data evaluation. Two or four missing runs in the 16-run design shown below significantly worsen the design’s ability to estimate factor effects.

One important issue is rising factor correlations when missing runs are present in the design. Figure 1 shows the factor correlation matrices for the same design with no missing runs, two missing runs, and four missing runs. If an off-diagonal element is shaded any color other than white, it means there is a correlation between two effects. The darker the shading, the stronger the correlation. While weak correlations of higher order terms can occur for optimal designs that don’t have missing runs, two missing runs will lead to more correlated factor effects that can be seen in the rising gray areas (as seen in the middle and left side of Figure 1). It might be tolerable since these correlations are not stronger than in the original design.

If there are four failed runs, much darker areas occur and some two-way interactions are confounded with each other, meaning that any model will be unable to fit both or distinguish between them (right side of Figure 1). This confounding and the strong factor correlations will likely have a negative impact on the model’s performance.

Figure 1: Comparison of factor correlations the design with no failed runs (left), two failed runs (middle), and four failed runs (right).

But what is the appropriate response when there are unmeasurable or failed runs? It might always be a good idea to analyze the data that is there. Imputing zeros for the missing runs would result in incorporating the missing runs into the modelling calculations, which might be not the best thing to do without any actual data for these points if we are unsure if the response is truly zero.

However, even treating these data points as missing will have quite a significant impact on model performance and accuracy which can be seen in the different model behaviors in Figure 2. A model built on the data with four failed runs greatly overestimates the yield with larger confidence bands in areas of the design space with missing runs and may skew modelling the influence of the flow rate compared to the model without missing runs. It might be a direct consequence of the stronger factor correlations and confounded factor effects discussed above. The impact of missing runs on the model performance can differ and is dependent on the number and position of the missing runs in the design. To be fair, one instead of four missing runs would have much less of a negative impact.

Figure 2: Model profiler and overlayed interaction comparison for models built without failed runs (top) and with four failed runs in the outer regions of temperature and time, which is treated as missing data (bottom).

What your missing runs are actually telling you

What caused the missing runs? If there were an error when conducting the first experiment, then the experiment can be repeated, and better results would be expected. In this example, high temperature and time settings caused the solution to overcook, while low settings failed to start a reaction. While repeating these experiments with the same settings would be pointless, they still offer valuable information. In the case of missing data due to combinations of factors resulting in failed runs (like in this example), then constraining the design space in the critical areas might be a good approach. Figure 3 shows an example that has only four augmented design points to constrain the design space in the critical areas, as well as the beneficial impact they have on the structure of the factor correlations. Testing the assumed extreme setting of a system before conducting the actual design using scoping designs might help to prevent this for future projects and save some time, resources and headaches for the scientists involved.

Figure 3: Four additional augmented runs for linear constraints that address the critical design space regions (left), resulting in a much less concerning correlation structure compared to the design with four missing runs in Figure 1 (right).

Summary

Missing runs don’t mean wasted data

Missing runs do not necessarily mean unusable results, but they will negatively influence the model’s performance, depending on their number and position. Asking why the runs were missing can provide additional understanding of the process and give some hints on how to proceed with the existing data, such as constraining the design space and augmenting some runs.

What a ‘failed’ DOE is really worth

Not every DOE is a success. No model is perfect. But there is valuable information in every planned and structured approach, such as DOE, even when the first set of experiments “failed” or don’t match your expectations. By using this information, every next step will lead to real process understanding that will ultimately deliver additional business value, decrease costs, and close projects successfully and on time.