Measuring inter-rater reliability for nominal data - which coefficients and confidence intervals are appropriate?
Standard
Measuring inter-rater reliability for nominal data - which coefficients and confidence intervals are appropriate? / Zapf, Antonia; Castell, Stefanie; Morawietz, Lars; Karch, André.
in: BMC MED RES METHODOL, Jahrgang 16, 05.08.2016, S. 93.Publikationen: SCORING: Beitrag in Fachzeitschrift/Zeitung › SCORING: Zeitschriftenaufsatz › Forschung › Begutachtung
Harvard
APA
Vancouver
Bibtex
}
RIS
TY - JOUR
T1 - Measuring inter-rater reliability for nominal data - which coefficients and confidence intervals are appropriate?
AU - Zapf, Antonia
AU - Castell, Stefanie
AU - Morawietz, Lars
AU - Karch, André
PY - 2016/8/5
Y1 - 2016/8/5
N2 - BACKGROUND: Reliability of measurements is a prerequisite of medical research. For nominal data, Fleiss' kappa (in the following labelled as Fleiss' K) and Krippendorff's alpha provide the highest flexibility of the available reliability measures with respect to number of raters and categories. Our aim was to investigate which measures and which confidence intervals provide the best statistical properties for the assessment of inter-rater reliability in different situations.METHODS: We performed a large simulation study to investigate the precision of the estimates for Fleiss' K and Krippendorff's alpha and to determine the empirical coverage probability of the corresponding confidence intervals (asymptotic for Fleiss' K and bootstrap for both measures). Furthermore, we compared measures and confidence intervals in a real world case study.RESULTS: Point estimates of Fleiss' K and Krippendorff's alpha did not differ from each other in all scenarios. In the case of missing data (completely at random), Krippendorff's alpha provided stable estimates, while the complete case analysis approach for Fleiss' K led to biased estimates. For shifted null hypotheses, the coverage probability of the asymptotic confidence interval for Fleiss' K was low, while the bootstrap confidence intervals for both measures provided a coverage probability close to the theoretical one.CONCLUSIONS: Fleiss' K and Krippendorff's alpha with bootstrap confidence intervals are equally suitable for the analysis of reliability of complete nominal data. The asymptotic confidence interval for Fleiss' K should not be used. In the case of missing data or data or higher than nominal order, Krippendorff's alpha is recommended. Together with this article, we provide an R-script for calculating Fleiss' K and Krippendorff's alpha and their corresponding bootstrap confidence intervals.
AB - BACKGROUND: Reliability of measurements is a prerequisite of medical research. For nominal data, Fleiss' kappa (in the following labelled as Fleiss' K) and Krippendorff's alpha provide the highest flexibility of the available reliability measures with respect to number of raters and categories. Our aim was to investigate which measures and which confidence intervals provide the best statistical properties for the assessment of inter-rater reliability in different situations.METHODS: We performed a large simulation study to investigate the precision of the estimates for Fleiss' K and Krippendorff's alpha and to determine the empirical coverage probability of the corresponding confidence intervals (asymptotic for Fleiss' K and bootstrap for both measures). Furthermore, we compared measures and confidence intervals in a real world case study.RESULTS: Point estimates of Fleiss' K and Krippendorff's alpha did not differ from each other in all scenarios. In the case of missing data (completely at random), Krippendorff's alpha provided stable estimates, while the complete case analysis approach for Fleiss' K led to biased estimates. For shifted null hypotheses, the coverage probability of the asymptotic confidence interval for Fleiss' K was low, while the bootstrap confidence intervals for both measures provided a coverage probability close to the theoretical one.CONCLUSIONS: Fleiss' K and Krippendorff's alpha with bootstrap confidence intervals are equally suitable for the analysis of reliability of complete nominal data. The asymptotic confidence interval for Fleiss' K should not be used. In the case of missing data or data or higher than nominal order, Krippendorff's alpha is recommended. Together with this article, we provide an R-script for calculating Fleiss' K and Krippendorff's alpha and their corresponding bootstrap confidence intervals.
KW - Algorithms
KW - Breast Neoplasms
KW - Confidence Intervals
KW - Data Interpretation, Statistical
KW - Female
KW - Humans
KW - Observer Variation
KW - Reproducibility of Results
KW - Retrospective Studies
KW - Journal Article
U2 - 10.1186/s12874-016-0200-9
DO - 10.1186/s12874-016-0200-9
M3 - SCORING: Journal article
C2 - 27495131
VL - 16
SP - 93
JO - BMC MED RES METHODOL
JF - BMC MED RES METHODOL
SN - 1471-2288
ER -