Measurement Systems Analysis

Style
section-padding-none

What is measurement systems analysis?

Measurement systems analysis (MSA) is the process of identifying, characterizing, and quantifying sources of measurement variability associated with a measurement system. The analysis can include any characteristic of the measurement process that the investigator believes may influence measured values such as instrument type, set-up procedures, operators, time, etc.

MSA response data can be quantitative (numerical) or qualitative (categorical). For qualitative measurements, the two main sources of variability are random and systematic error. Estimation of random error is performed by measuring the same quantity multiple times under the same conditions (repeatability) and for differing conditions (reproducibility). The estimate of systematic error is often performed by taking multiple measurements on a known quantity and comparing the results to the known (or verified) value for that quantity. Bias and linearity are characteristics associated with systematic error.

What are the benefits of using measurement systems analysis?

MSA provides a method to better understand the quality of measurements being made. If I am unsure that my measurement system can accurately and repeatably quantify a given characteristic, why use that system? Insomuch as decisions are made based on measured values, it is foundational to many processes. MSA allows me to quantify the closeness and variability of my measurements.

$ICC = \frac{\hat{\sigma}^2_{\text{process}}}{\left( \hat{\sigma}^2_{\text{process}} + \hat{\sigma}^2_{\text{repeatability}} \right)} = \frac{1}{0.2213 + 1} = 0.8188$

$\frac{P}{T} = k \frac{\hat{\sigma}_{\text{repeatability}}}{\text{USL} - \text{LSL}} = 6 \times \frac{0.4705}{10} = 0.2823$

What is a gauge repeatability and reproducibility study?

A gauge repeatability and reproducibility (GR&R) study can be considered part of an MSA where the focus is on a single instrument. An MSA conducted on qualitative outcomes focuses on metrics such as interrater/intrarater reliability, test-retest reliability, and misclassification probabilities. They are sometimes called attribute gauge studies.

Gauge R&R study example: Estimation of random error components for a single instrument

An automated measurement tool is used to measure parts coming off a manufacturing line. A single part from the center of the process distribution is created for the study. It is assumed that the part does not change over the course of the study. Early on Day 1, the part is measured three times in rapid succession. Later in the day, the part is measured three more times by the same operator. This process is repeated every other day four more times. The data are shown below. Rep indicates measurements made in rapid succession, while Run corresponds to a set of replications.

For the analysis, the variance component (VC) associated with Rep is used to estimate repeatability. The Run and Day VCs, along with the Rep VC, are used to estimate reproducibility.

Day

Run

Rep

Y

1
1
1
112.3
1
1
2
112.6
1
1
3
112.5
1
2
1
112
1
2
2
112
1
2
3
111.9
2
3
1
112
2
3
2
112.2
2
3
3
112.2
2
4
1
111.6
2
4
2
111.6
2
4
3
111.4
3
5
1
112.8
3
5
2
112.9
3
5
3
112.7
3
6
1
112.5
3
6
2
112.6
3
6
3
112.6
4
7
1
113.2
4
7
2
113.1
4
7
3
113.1
4
8
1
112.5
4
8
2
112.8
4
8
3
112.6
5
9
1
112.3
5
9
2
112.4
5
9
3
112.5
5
10
1
112.6
5
10
2
112.4
5
10
3
112.5

Variance components results are shown below.

Component

VC

% of Total

Sqrt(VC)

Day
0.1213
54.8
0.3483
Run[Day]
0.0887
40.1
0.2978
Repeatability
0.0113
5.1
0.1065
Total
0.2213
100.0
0.4705

The specification range (upper specification limit [USL] - lower specification limit [LSL]) for a part is ±5 units. The process variance is 1 unit2.  The intraclass correlation (ICC) for the measurement system is

$ICC = \frac{\sigma^2_{\text{process}}}{\left( \sigma^2_{\text{process}} + \sigma^2_{\text{repeatability}} \right)} = \frac{1}{0.2213 + 1} = 0.8188$

ICC measures the proportion of total variance attributable to the process. Given that it is favorable for the measurement system to account for a small proportion of the total variance, higher values are better.

The precision-to-tolerance ratio, based on the AIAG MSA manual guidance, is

$\frac{P}{T} = k \frac{\hat{\sigma}_{\text{repeatability}}}{\text{USL} - \text{LSL}} = 6 \times \frac{0.4705}{10} = 0.2823$

Estimation of bias example

Five parts evenly spanning the process range are created for the study. It is assumed that the parts do not change over the course of the study. Using the same automated measurement tool, a part is selected randomly, and three measurements are taken in quick succession. This is repeated until all parts are selected. After the study, the parts are measured with a second method, which is more accurate but destructive. Results are shown below. Y corresponds to the measurements taken during the bias study; Y2 corresponds to the destructive measurements. Bias is the difference between Y and Y2.

Part

Rep

Y

Y2

Bias

4
1
112.4
110
2.4
4
2
112.1
110
2.1
4
3
112.1
110
2.1
3
1
101.2
100
1.2
3
2
101.4
100
1.4
3
3
101.3
100
1.3
2
1
90.4
90
0.4
2
2
90.4
90
0.4
2
3
90.5
90
0.5
1
1
80.1
80
0.1
1
2
79.8
80
-0.2
1
3
80
80
0
5
1
123.9
120
3.9
5
2
124.2
120
4.2
5
3
124.1
120
4.1

The average bias for each part is shown below. An ANOVA indicates that all measurements, except the those taken on Part 1, are significantly different from their nominal value.

Part

Nominal Value

Average Measurement

Average Bias

p Value

1
80
79.967
-0.033
0.7418
2
90
90.433
0.433
0.0059
3
100
101.300
1.300
0.0020
4
110
112.200
2.211
0.0021
5
120
124.067
4.067
0.0005

A plot of the bias as a function of the nominal value and a test for non-significance of the slope indicate the tool is exhibiting non-linearity.

What is method validation?

In life sciences, particularly in assay development, the process of quantifying measurement system variability is part of method validation. Analytic method validation  found in chemical or biological assay development shares quantitative methodologies with MSA, in particular the estimation of repeatability and reproducibility.

Industry standards and guidelines

Several industry guidance groups and standards organizations – such as the American Society for Testing and Measurement  (ASTM), Automotive Industry Action Group (AIAG), International Council for Harmonization of Technical Requirements for Pharmaceutical for Human Use (ICH), International Organization for Standardization (ISO), and Joint Committee for Guides in Metrology (JCGM) – have developed detailed definitions and methodologies around MSA. A partial list of definitions can be found under the Terminology heading below. A list of standards and guidelines can be found in the Standards and Guidelines section. For a more in depth understanding of MSA, see the References page.

Terminology

Definitions are based on the International vocabulary of metrology (JCGM 200:2012).

Standards and guidelines

Two seminal documents in the development of terminology and methodologies used in metrology are the Guide to the expression of uncertainty in measurement (GUM) (JCGM 100:2008) and the International vocabulary of basic and general terms in metrology (VIM) (JCGM 200:2012), maintained by the Joint Committee for Guides in Metrology (JCGM). Standards organization and industry guidance groups such as the International Organization for Standards (ISO), the American Society for Testing and Materials (ASTM), and the Automotive Industry Action Group (AIAG), have developed guidelines for conducting MSAs.

Note: The JCGM is a collaborative effort between eight organizations: Bureau International des Poids et Mesures (BIPM), Innovative Engineering Consortium (IEC), International Federation of Clinical Chemistry and Laboratory Medicine (IFCC), International Laboratory Accreditation Cooperation (ILAC), International Organization for Standardization (ISO), International Union of Pure and Applied Chemistry (IUPAC), International Union of Pure and Applied Physics (IUPAP), and International Organization of Legal Metrology (OIML).

References

Burdick, Richard K.; Borror, Connie M.; Montgomery, Douglas C. (2005). Design and Analysis of Gauge R&R Studies: Making Decisions with Confidence Intervals in Random and Mixed ANOVA Models. SIAM.

Eisenhart, C. (1963) Realistic Evaluation of the Precision and Accuracy of Instrument Calibration Systems, Journal of Research of the National Bureau of Standards – C, Engineering and Instrumentation, 67C(2).

Montgomery, D.C. & Runger G. C. (1993a), “Gauge Capability and Designed Experiments. Part I: Basic Methods”, Quality Engineering, 6(1), 115–135.

Montgomery, D.C. & Runger G. C. (1993b), “Gauge Capability and Designed Experiments. Part II: Experimental Design Models and Variance Component Estimation”, Quality Engineering, 6(2), 289–305.

Sandler, L.C. (2020, July 7). Analytical Method Validation [video]. National Institute of Standards and Technology.  Analytical Method Validation

Wheeler, Donald (2006). EMP III: Evaluating the Measurement Process & Using Imperfect Data. SPC Press.

Wheeler, D. J. (2006). EMP III Using Imperfect Data. Knoxville, TN: SPC Press.

layout
2 column
Style
columns-75-25, section-top-padding-xsmall