Example of a Covering Array
In this example, you use a covering array design to test software and operation system compatibility. You are interested in finding out which combinations of factors are likely to cause failures. There are four factors of interest:
• Web Browser (Safari, IE, Firefox, Chrome, Other)
• Operating System (Windows or Apple macOS)
• RAM (4, 8, or 16 MB)
• Connection Speed at three settings (0-1, 1-5, or greater than 5 Mbps)
The response is whether the system functions properly for each combination of factor settings.
Testing each combination of settings would require 90 (5 × 2 × 3 × 3) trials. To keep the run size manageable, you decide to require Strength 3 coverage, indicating that all combinations of any three factors are tested.
Create the Design
Create the Strength 3 covering array by following these steps.
1. Select Help > Sample Data Folder and open Design Experiment/Software Factors.jmp.
The Software Factors.jmp data table contains the factors and their settings.
2. Select DOE > Special Purpose > Covering Array.
3. From the menu next to Strength: t = , select 3.
4. Click the Covering Array red triangle, select Load Factors. This opens a window that enables you to select a factors table.
5. Select the Software Factors.jmp data table.
The Factors section is populated with the four factors and their levels.
Figure 20.4 Factors for Software Factors
6. Click Continue.
The Restrict Factor Level Combinations section opens, where you can enter restrictions on the design settings. Because there are no restrictions for this design, do not change the default selection of None.
7. Click Make Design.
The Design section opens to show a 45-run design.
Figure 20.5 Design and Metrics for Software Factors
In the Metrics section, consider the row that corresponds to t = 3. The Coverage is 100%, indicating that the design covers 100% of the three-factor interactions. This is what you want, because you requested a Strength 3 design. For t = 3, the Diversity column indicates that 68.33% of the three-factor interactions that appear are distinct. There is some minor repetition of three-factor combinations.
For t = 4, the Coverage is 50%, indicating that the design covers half of the four-factor interactions. There are 90 possible distinct combinations of the four factor settings. The 45 runs in the design comprise one-half of these distinct combinations. The Diversity value of 100% reinforces the fact that none of the four-way interactions are repeated.
8. Click Make Table.
Analyze the Experimental Data
Now that you have your design table, you can conduct your experiment and record your data in the Response column of the design table. Example experimental results are in the Software Data.jmp sample data table.
1. Select Help > Sample Data Folder and open Design Experiment/Software Data.jmp.
2. In the Table panel, click the green triangle next to the Analysis script.
Figure 20.6 Analysis of Software Experimental Data
The Summary section indicates that three tests failed and four tests did not result in a pass or fail outcome.
The Failure Analysis Details section gives a breakdown of failures in terms of the associated three-way interactions. The section lists only combinations of factor levels where all tests resulted in failure. If any test that involves a given three-way combination of settings results in success, then that three-way combination of settings cannot be responsible for system failure.
Two failures were associated with Web Browser set to Firefox, Operating System set to Apple macOS, and RAM set to 8 MB. Notice that this combination led to failure regardless of the setting of Connection Speed.
3. Select any row in the 3 Factor Interactions report to select the corresponding rows and columns in the data table.