Statistical engineering to reduce variability in vaccine production

ANNE MILLEY: Can you please tell us about the case study you co-authored on statistical engineering to stabilize vaccine supply? In particular, how it helped you identify root cause and effective corrective action. Maybe that's part of what you already shared a little bit of, but since you wrote that paper, I wanted to call that out.

JULIA O'NEILL: Yes, this was another very interesting large team effort, and you probably saw in that paper, there were many authors on the paper. It was a team I led where we were experiencing unexplained shifts in a quality characteristic of a vaccine. What complicates a lot of these biotech processes is that the testing process itself is almost as lengthy and complex as the production process. In this particular case, the analytical testing took about two weeks to complete, involved many plates that had been plated with cells, and then the cells are grown. They're exposed to some of the vaccine product, and then they're incubated a while, and then we count how many plaques are formed.

You can just picture the variability that occurs in that situation. There are a lot of things put in place to control or correct for the variability, but the process is so complex. I mean, you think about a biological process that takes a couple of weeks to complete —even looking at changes in the reagents that are used through those two weeks of testing, it's very difficult to pinpoint what might cause a shift in results. So we used a number of statistical methods.

The first one that gave us a big breakthrough was a simple CUSUM. It was a CUSUM overlay, where we said, well, we produce this vaccine at multiple sites. We test it in multiple laboratories, and we test it at several points in the process from an intermediate through to the final product. So let's use CUSUM to see, when did we actually see these shifts in the attribute? And did any of them occur at the same time?

Very simple, but man, I remember showing that graph, and it was so clear instantly. It pointed to the laboratory. Just I mean one of those one-minute glance at it, and I didn't have to do much talking.

So I used CUSUM extensively. And that was one of the biggest impacts it had. But then from that point on, there was still a lot of work remaining to track down exactly where within the laboratory did this occur. This kicked off extensive lean labs' work to bring down variability, because this was an assay result, which is used for critical decision making and affected the supply chain.

As a result of all that work, the statistical engineering aspect is that we put in place some very focused monitoring tools for performance of certain steps in the assay. The laboratory, actually, through their Six Sigma kind of Lean efforts, brought the basic level variability in that result down from—for years, it had been about 40%—they brought it down to 21%, and it has been rock solid there. I'm quite sure it's still there. Occasionally, I run into people who still work in that lab, and they say it's still there. So amazing reduction in variability, but then, also, the monitoring tools that we put in place to make sure that never drifts again.

ANNE MILLEY: Well, it's so important—there are places that you have to get good measurements.

JULIA O'NEILL: Yeah, and boy, the devil's in the details for a lot of things, but this is one area where it can get down to the most granular information, and statistics helps you dive down but also make those connections.

ANNE MILLEY: And see what's important and what you need to do as a next step, and then the benefits you get to realize. That's amazing. Well, Julia, thank you so much for taking the time to share your experience and expertise with our audience, and I've really enjoyed talking with you about this.

JULIA O'NEILL: Well, I have, too. Thank you for inviting me.

ANNE MILLEY: If you are looking to simplify the regulatory compliance process, watch our webcast on pharmaceutical product stability. Find it at And come back to to see more interviews on analytics topics. Thanks, and goodbye.