A prediction-based method to estimate student learning outcome: Impact of response rate and gender differences on evaluation results

TY - JOUR

T1 - A prediction-based method to estimate student learning outcome: Impact of response rate and gender differences on evaluation results

AU - Grebener, Binia-Laureen

AU - Barth, Janina

AU - Anders, Sven

AU - Beißbarth, Tim

AU - Raupach, Tobias

PY - 2021/5

Y1 - 2021/5

N2 - BACKGROUND: Low response rates threaten the reliability and validity of student evaluations of teaching. Previous research has shown that asking students to predict how satisfied their fellow students were with a course produces reliable results at lower response rates. The aim of this study was to investigate whether this prediction-based method can also be used to evaluate student learning outcome.METHODS: Before and after a cardiorespiratory module, 128 fourth-year medical students provided self-assessments and predictions of performance on 27 specific learning objectives and took formative tests on the respective contents. Pre-post performance gain was compared across all three modalities.RESULTS: Formative exam results indicated a performance gain of 63.0%. Self-assessed and prediction-based performance gains were identical (67.8%) but both slightly overestimated actual performance gain. Irrespective of the method used, a response rate of 20% was sufficient to produce reliable results. Compared to male students, females greatly overestimated their peers' performance which led to inflated performance gain values.CONCLUSIONS: Student self-assessments and predictions are equally valid sources of learning outcome measures, and low response rates are sufficient to produce stable results. When using a prediction-based approach, a tendency to overestimate learning outcome in female students needs to be taken into account.

AB - BACKGROUND: Low response rates threaten the reliability and validity of student evaluations of teaching. Previous research has shown that asking students to predict how satisfied their fellow students were with a course produces reliable results at lower response rates. The aim of this study was to investigate whether this prediction-based method can also be used to evaluate student learning outcome.METHODS: Before and after a cardiorespiratory module, 128 fourth-year medical students provided self-assessments and predictions of performance on 27 specific learning objectives and took formative tests on the respective contents. Pre-post performance gain was compared across all three modalities.RESULTS: Formative exam results indicated a performance gain of 63.0%. Self-assessed and prediction-based performance gains were identical (67.8%) but both slightly overestimated actual performance gain. Irrespective of the method used, a response rate of 20% was sufficient to produce reliable results. Compared to male students, females greatly overestimated their peers' performance which led to inflated performance gain values.CONCLUSIONS: Student self-assessments and predictions are equally valid sources of learning outcome measures, and low response rates are sufficient to produce stable results. When using a prediction-based approach, a tendency to overestimate learning outcome in female students needs to be taken into account.

U2 - 10.1080/0142159X.2020.1867714

DO - 10.1080/0142159X.2020.1867714

M3 - SCORING: Journal article

C2 - 33502287

VL - 43

SP - 524

EP - 530

JO - MED TEACH

JF - MED TEACH

SN - 0142-159X

IS - 5

ER -