The doctrines of design of experiments

1. Thou shalt not use OFAT (one factor at a time)

“For the path to enlightenment lies in interaction, not isolation.”

This is the big epiphany when you first learn about DOE. Most scientists and engineers are indoctrinated from an early age with the idea that you can only change one thing at a time in your experiments. It’s not completely wrong, and they cling to this reassuring logic as an article of faith in the scientific method. Then they are challenged by having to optimise a system in the wicked world of industrial R&D, and they find out that one-dimensional experiments don’t work for multidimensional problems. The fear of multifactor interactions is the beginning of DOE wisdom.

2. Replication is sacred

“Only through repetition shall the truth be revealed.”

Experimenters have strange superstitions about “noise.” Biologists are so scared of the randomness of living systems that they do everything in triplicate, while chemists are offended when you suggest they might get a different result from a repeated experiment. Replication in your experiments reveals the true signal and delivers us from the temptation of chasing noise. Puritan followers of R.A. Fisher will say you need exact repeats of treatments to estimate pure error. But there is also replication of each level of each factor in a balanced factorial design.

3. Randomization is divine protection

“Let chance guard thee from bias.”

Randomizing the order of runs protects against the false prophets of lurking variables and confounding effects. It’s the shield against hidden influences. Followers of the new Bayesian Optimization cult cast out randomization in the pursuit of efficiency. But they do so at their peril. When changes to factors track with system drift, the divine causal clarity we seek is impossible without god-like omniscience. The newer method is therefore more suited to improving than proving. Fun fact: the “Bayesian” name comes from the theorem of the Reverend Thomas Bayes (1701-1761), statistician and Presbyterian minister of the Mount Sion Chapel, Kent, England

4. Center points are the oracle of curvature

“To truly know thy system, one must center thy system”

Here’s one where there are bitter differences of opinion between different sects. Orthodox procentropuntalists offer up repeated center point runs as a sacrifice to the gods at the altar of experimentation. They preach that these are essential tests of curvature and pure estimates of error. The more enlightened contracentropuntalists understand that this is an arcane interpretation of the DOE scriptures. The center is the least interesting part of the factor space, so multiple runs at the midpoint of all factors feels like a penance. Using a more virtuous three-level design, such as a definitive screening design, can reveal which factor or factors are responsible for any curvature.

5. Always seek parsimony and respect the hierarchy of effects

“For in the end, all models are wrong”

When building models from our experimental data we follow the sage advice of 14th century philosopher and theologian William of Ockham: Entia non sunt multiplicanda praeter necessitatem, which translates as, "Entities must not be multiplied beyond necessity." That is, simple wins. (I feel like Bill should have listened to his own advice when wording his famous “razor.”)

The tenets of effect hierarchy and heredity guide us in this quest for parsimony: higher order effects are less likely to be active, and we only include in our models those higher order effects that contain active lower order effects. While the active/inactive dichotomy can feel crude and dogmatic, there is nevertheless good empirical support. In any case, the idea that a perfect model is attainable in the mortal realm is heretical to our creed. This is why you will so often hear evangelists quoting from the gospel of DOE guru George EP Box: “All models are wrong, but some are useful.” So profound!