{"id":835,"date":"2015-08-10T15:34:31","date_gmt":"2015-08-10T14:34:31","guid":{"rendered":"http:\/\/pcool.dyndns.org:8080\/statsbook\/?page_id=835"},"modified":"2025-07-04T13:44:04","modified_gmt":"2025-07-04T12:44:04","slug":"repeatability-and-reproducibility","status":"publish","type":"page","link":"https:\/\/pcool.dyndns.org\/index.php\/repeatability-and-reproducibility\/","title":{"rendered":"Repeatability and Reproducibility"},"content":{"rendered":"\n

If a measurement is repeated, it is likely the measurement value is different each time. If the variation is small, the repeatability is high. Obviously, there is a difference if the measurement is repeated by the same person or another person. Intra observer variation<\/a><\/em><\/strong> is the variation that occurs when the same persons repeats the measurements. Whilst Inter observer variation<\/a><\/em><\/strong> is the variation that occurs when a different person repeats the measurements. Normally, the inter-observer variation is larger than the intra-observer variation.<\/p>\n\n\n\n

The American Society for Testing and Materials (ASTM) has defined repeatability and reproducibility1<\/a><\/sup>.<\/p>\n\n\n\n

Please recall<\/a> precision is the reciprocal of the variance.<\/em><\/strong><\/p>\n\n\n\n

Repeatability<\/strong><\/p>\n\n\n\n

Repeatability is precision<\/strong> determined from multiple tests done under repeatability conditions<\/strong>: the test is conducted by the same operator, using the same equipment and laboratory within a short period of time so that neither the equipment nor the environment is likely to change significantly.<\/p>\n\n\n\n

Reproducibility<\/strong><\/p>\n\n\n\n

Reproducibility is precision<\/strong> determined from multiple tests done under reproducibility conditions<\/strong>: the test is conducted in different laboratories with different operators and different environmental conditions.<\/p>\n\n\n\n

When an accepted reference value is known, the bias<\/a> can be expressed.<\/p>\n\n\n\n

The precision limits <\/strong>of the repeatability and reproducibility can be calculated from the sample standard deviation of the test results. The number of tests should be at least 30, so that the sample standard deviation is a reasonable estimate of the population standard deviation. The repeatability precision limit (r) and the reproducibility precision limit (R) are useful for comparing test results within and between laboratories. They are calculated by multiplying the repeatability standard deviation (sr) or the reproducibility standard deviation (sR) by 2.8 respectively. The factor 2.8 is derived from 1.96 (95% of the population is within 1.96 standard deviations of the mean) times the square root of 2.<\/p>\n\n\n\n

For example, consider the measurements of femoral heads in the heads.rda<\/a> file. The data frame is called heads and there are five variables: number, the accepted reference value (measurement of the femoral head using callipers at the time of excision at total hip replacement) and three radiological measurements performed before the surgery. Measurements m1 and m2 were done under repeatability conditions and measurement m3 was done under reproducibility conditions.<\/p>\n\n\n\n

Load the data file in R and check the data frame:<\/p>\n\n\n\n

heads\n<\/mark>   number reference m1 m2 m3\n1       1        52 54 55 53\n2       2        50 50 49 56\n3       3        50 51 47 52\n4       4        52 53 53 53\n5       5        52 50 51 52\n6       6        48 49 48 51\n7       7        55 53 56 56\n8       8        55 52 55 59\n9       9        53 54 54 53\n10     10        48 47 49 52\n11     11        50 51 48 54\n12     12        48 47 47 53\n13     13        50 49 51 50\n14     14        49 50 49 55\n15     15        52 51 51 52\n16     16        51 53 52 50\n17     17        51 50 49 53\n18     18        51 52 51 56\n19     19        50 51 52 53\n20     20        54 56 55 56\n21     21        49 48 48 53\n22     22        50 51 51 55\n23     23        53 52 52 55\n24     24        48 48 50 52\n25     25        52 52 53 55\n26     26        51 51 51 56\n27     27        55 53 55 59\n28     28        50 53 49 52\n29     29        54 52 55 57\n30     30        50 52 49 59<\/mark><\/em><\/code><\/pre>\n\n\n\n
As there is an accepted reference value (the calliper measurements), first calculate the biases of the measurements:<\/p>\n\n\n\n
bias1 <- heads$m1 - heads$reference<\/span><\/em>\nbias2 <- heads$m2 - heads$reference<\/span><\/em>\nbias3 <- heads$m3 - heads$reference<\/span><\/em><\/code><\/pre>\n\n\n\n
Are the biases Normally distributed<\/a>? This can be checked with the Shapiro-Wilks test for Normality<\/a>:<\/p>\n\n\n\n
shapiro.test(bias1)<\/mark>\n\tShapiro-Wilk normality test\n\ndata:  bias1\nW = 0.94217, p-value = 0.1041<\/mark>\n<\/mark><\/em>\nshapiro.test(bias2)<\/mark>\n\tShapiro-Wilk normality test\n\ndata:  bias2\nW = 0.94215, p-value = 0.1039<\/mark>\n\nshapiro.test(bias3)\n<\/mark>\tShapiro-Wilk normality test\n\ndata:  bias3\nW = 0.96116, p-value = 0.3315<\/mark><\/em><\/code><\/pre>\n\n\n\n
All three tests are non significant. It can therefore be concluded that it is reasonable to assume a Normal distribution as a model for the data.<\/p>\n\n\n\n
Next, check if there is bias by performing a t-test (one sample two sided):<\/p>\n\n\n\n
t.test(bias1)\n<\/mark>\tOne Sample t-test\n\ndata:  bias1\nt = 0.24229, df = 29, p-value = 0.8103<\/strong>\nalternative hypothesis: true mean is not equal to 0\n95 percent confidence interval:\n -0.4960831  0.6294164\nsample estimates:\n mean of x \n0.06666667 <\/mark>\n\nt.test(bias2)\n<\/mark>\tOne Sample t-test\n\ndata:  bias2\nt = 0.27304, df = 29, p-value = 0.7868<\/strong>\nalternative hypothesis: true mean is not equal to 0\n95 percent confidence interval:\n -0.4327081  0.5660415\nsample estimates:\n mean of x \n0.06666667 <\/mark>\n\nt.test(bias3)\n<\/mark>\tOne Sample t-test\n\ndata:  bias3\nt = 7.2677, df = 29, p-value = 5.285e-08<\/strong>\nalternative hypothesis: true mean is not equal to 0\n95 percent confidence interval:\n 2.131801 3.801532\nsample estimates:\nmean of x \n 2.966667 <\/mark><\/em><\/code><\/pre>\n\n\n\n
The biases of m1 (bias1) and m2 (bias2), performed under repeatability conditions are not significantly different from zero (p=0.81 and p=0.79 respectively). So the repeatability measurements are unbiased. However, the bias in the measurements performed under reproducibility conditions are significantly different from zero (p=0.00000005). So, the reproducibility measurements are biased by:<\/p>\n\n\n\n
mean(bias3)<\/em><\/span>\n[1] 2.966667<\/em><\/span><\/code><\/pre>\n\n\n\n
or 3 mm.<\/p>\n\n\n\n
To express the repeatability<\/strong> according to the ASTM standard2<\/a><\/sup>:<\/p>\n\n\n\n
Subtract measurement m2 from m1 and call this ‘rep’:<\/p>\n\n\n\n
rep<-heads$m1-heads$m2<\/em><\/span>\nrep<\/em><\/span>\n [1] -1  1  4  0 -1  1 -3 -3  0 -2  3  0 -2  1  0  1  1  1 -1  1  0  0  0 -2 -1  0 -2  4 -3  3<\/em><\/span><\/code><\/pre>\n\n\n\n
So, the repeatability standard deviation (sr) is:<\/p>\n\n\n\n
sr<-sd(rep)<\/em><\/span>\nsr<\/em><\/span>\n[1] 1.893728<\/em><\/span><\/code><\/pre>\n\n\n\n
and the repeatability (r) is:<\/p>\n\n\n\n
repeatability<-qnorm(0.975)*sqrt(2)*sr<\/em><\/span>\nrepeatability<\/em><\/span>\n[1] 5.249051<\/em><\/span><\/code><\/pre>\n\n\n\n
or 5.2 mm.<\/p>\n\n\n\n
Please note as per standard describe above:<\/p>\n\n\n\n
qnorm(0.975)*sqrt(2)\n<\/mark>[1] 2.771808<\/mark><\/em><\/code><\/pre>\n\n\n\n
To express the reproducibility<\/strong> according the the ASTM standard3<\/a><\/sup>:<\/p>\n\n\n\n
Subtract measurement m3 from m1 and call this ‘repro’:<\/p>\n\n\n\n
repro<-heads$m1-heads$m3<\/em><\/span>\nrepro<\/em><\/span>\n [1]  1 -6 -1  0 -2 -2 -3 -7  1 -5 -3 -6 -1 -5 -1  3 -3 -4 -2  0 -5 -4 -3 -4 -3 -5 -6  1 -5 -7<\/em><\/span><\/code><\/pre>\n\n\n\n
So, the reproducibility standard deviation (sR) is:<\/p>\n\n\n\n
sR<-sd(repro)<\/em><\/span>\nsR<\/em><\/span>\n[1] 2.61758<\/em><\/span><\/code><\/pre>\n\n\n\n
and the reproducibility R is:<\/p>\n\n\n\n
reproducibility<-qnorm(0.975)*sqrt(2)*sR\nreproducibility\n[1] 7.255428<\/span><\/em><\/span><\/code><\/pre>\n\n\n\n
or 7.3 mm.<\/p>\n","protected":false},"excerpt":{"rendered":"
If a measurement is repeated, it is likely the measurement value is different each time. If the variation is small, the repeatability is high. Obviously, there is a difference if the measurement is repeated by the same person or another person. Intra observer variation is the variation that occurs when the same persons repeats the […]<\/p>\n","protected":false},"author":2,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"inline_featured_image":false,"footnotes":""},"class_list":["post-835","page","type-page","status-publish","hentry"],"_links":{"self":[{"href":"https:\/\/pcool.dyndns.org\/index.php\/wp-json\/wp\/v2\/pages\/835","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/pcool.dyndns.org\/index.php\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/pcool.dyndns.org\/index.php\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/pcool.dyndns.org\/index.php\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/pcool.dyndns.org\/index.php\/wp-json\/wp\/v2\/comments?post=835"}],"version-history":[{"count":4,"href":"https:\/\/pcool.dyndns.org\/index.php\/wp-json\/wp\/v2\/pages\/835\/revisions"}],"predecessor-version":[{"id":4896,"href":"https:\/\/pcool.dyndns.org\/index.php\/wp-json\/wp\/v2\/pages\/835\/revisions\/4896"}],"wp:attachment":[{"href":"https:\/\/pcool.dyndns.org\/index.php\/wp-json\/wp\/v2\/media?parent=835"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}