Development and evaluation of a standardized peer-training in the context of peer review for quality assurance in work capacity evaluation
Standard
Development and evaluation of a standardized peer-training in the context of peer review for quality assurance in work capacity evaluation. / Strahl, André; Gerlich, Christian; Alpers, Georg W; Ehrmann, Katja; Gehrke, Jörg; Müller-Garnn, Annette; Vogel, Heiner.
In: BMC MED EDUC, Vol. 18, No. 1, 13.06.2018, p. 135.Research output: SCORING: Contribution to journal › SCORING: Journal article › Research › peer-review
Harvard
APA
Vancouver
Bibtex
}
RIS
TY - JOUR
T1 - Development and evaluation of a standardized peer-training in the context of peer review for quality assurance in work capacity evaluation
AU - Strahl, André
AU - Gerlich, Christian
AU - Alpers, Georg W
AU - Ehrmann, Katja
AU - Gehrke, Jörg
AU - Müller-Garnn, Annette
AU - Vogel, Heiner
PY - 2018/6/13
Y1 - 2018/6/13
N2 - BACKGROUND: The German quality assurance programme for evaluating work capacity is based on peer review that evaluates the quality of medical experts' reports. Low reliability is thought to be due to systematic differences among peers. For this purpose, we developed a curriculum for a standardized peer-training (SPT). This study investigates, whether the SPT increases the inter-rater reliability of social medical physicians participating in a cross-institutional peer review.METHODS: Forty physicians from 16 regional German Pension Insurances were subjected to SPT. The three-day training course consist of nine educational objectives recorded in a training manual. The SPT is split into a basic module providing basic information about the peer review and an advanced module for small groups of up to 12 peers training peer review using medical reports. Feasibility was tested by assessing selection, comprehensibility and subjective use of contents delivered, the trainers' delivery and design of training materials. The effectiveness of SPT was determined by evaluating peer concordance using three anonymised medical reports assessed by each peer. Percentage agreement and Fleiss' kappa (κm) were calculated. Concordance was compared with review results from a previous unstructured, non-standardized peer-training programme (control condition) performed by 19 peers from 12 German Pension Insurances departments. The control condition focused exclusively on the application of peer review in small groups. No specifically training materials, methods and trainer instructions were used.RESULTS: Peer-training was shown to be feasible. The level of subjective confidence in handling the peer review instrument varied between 70 and 90%. Average percentage agreement for the main outcome criterion was 60.2%, resulting in a κm of 0.39. By comparison, the average percentage concordance was 40.2% and the κm was 0.12 for the control condition.CONCLUSION: Concordance with the main criterion was relevant but not significant (p = 0.2) higher for SPT than for the control condition. Fleiss' kappa coefficient showed that peer concordance was higher for SPT than randomly expected. Nevertheless, a score of 0.39 for the main criterion indicated only fair inter-rater reliability, considerably lower than the conventional standard of 0.7 for adequate reliability.
AB - BACKGROUND: The German quality assurance programme for evaluating work capacity is based on peer review that evaluates the quality of medical experts' reports. Low reliability is thought to be due to systematic differences among peers. For this purpose, we developed a curriculum for a standardized peer-training (SPT). This study investigates, whether the SPT increases the inter-rater reliability of social medical physicians participating in a cross-institutional peer review.METHODS: Forty physicians from 16 regional German Pension Insurances were subjected to SPT. The three-day training course consist of nine educational objectives recorded in a training manual. The SPT is split into a basic module providing basic information about the peer review and an advanced module for small groups of up to 12 peers training peer review using medical reports. Feasibility was tested by assessing selection, comprehensibility and subjective use of contents delivered, the trainers' delivery and design of training materials. The effectiveness of SPT was determined by evaluating peer concordance using three anonymised medical reports assessed by each peer. Percentage agreement and Fleiss' kappa (κm) were calculated. Concordance was compared with review results from a previous unstructured, non-standardized peer-training programme (control condition) performed by 19 peers from 12 German Pension Insurances departments. The control condition focused exclusively on the application of peer review in small groups. No specifically training materials, methods and trainer instructions were used.RESULTS: Peer-training was shown to be feasible. The level of subjective confidence in handling the peer review instrument varied between 70 and 90%. Average percentage agreement for the main outcome criterion was 60.2%, resulting in a κm of 0.39. By comparison, the average percentage concordance was 40.2% and the κm was 0.12 for the control condition.CONCLUSION: Concordance with the main criterion was relevant but not significant (p = 0.2) higher for SPT than for the control condition. Fleiss' kappa coefficient showed that peer concordance was higher for SPT than randomly expected. Nevertheless, a score of 0.39 for the main criterion indicated only fair inter-rater reliability, considerably lower than the conventional standard of 0.7 for adequate reliability.
KW - Curriculum
KW - Feasibility Studies
KW - Germany
KW - Humans
KW - Medical Staff
KW - Observer Variation
KW - Peer Group
KW - Peer Review
KW - Physicians, Family
KW - Quality Assurance, Health Care
KW - Reproducibility of Results
KW - Work Capacity Evaluation
KW - Journal Article
U2 - 10.1186/s12909-018-1233-z
DO - 10.1186/s12909-018-1233-z
M3 - SCORING: Journal article
C2 - 29895284
VL - 18
SP - 135
JO - BMC MED EDUC
JF - BMC MED EDUC
SN - 1472-6920
IS - 1
ER -