Are Exam Questions Known in Advance? Using Local Dependence to Detect Cheating

TY - JOUR

T1 - Are Exam Questions Known in Advance? Using Local Dependence to Detect Cheating

AU - Zimmermann, Stefan

AU - Klusmann, Dietrich

AU - Hampe, Wolfgang

PY - 2016

Y1 - 2016

N2 - Cheating is a common phenomenon in high stakes admission, licensing and university exams and threatens their validity. To detect if some exam questions had been affected by cheating, we simulated how data would look like if some test takers possessed item preknowledge: Responses to a small number of items were set to correct for 1-10% of test takers. Item difficulty, item discrimination, item fit, and local dependence were computed using an IRT 2PL model. Then changes in these item properties from the non-compromised to the compromised dataset were scrutinized for their sensitivity to item preknowledge. A decline in the discrimination parameter compared with previous test versions and an increase in local item dependence turned out to be the most sensitive indicators of item preknowledge. A multiplicative combination of shifts in item discrimination, item difficulty, and local item dependence detected item preknowledge with a sensitivity of 1.0 and a specificity of .95 if 11 of 80 items were preknown to 10% of the test takers. Cheating groups smaller than 5% of the test takers were not detected reliably. In the discussion, we outline an effective search for items affected by cheating, which would enable faculty staff without IRT knowledge to detect compromised items and exclude them from scoring.

AB - Cheating is a common phenomenon in high stakes admission, licensing and university exams and threatens their validity. To detect if some exam questions had been affected by cheating, we simulated how data would look like if some test takers possessed item preknowledge: Responses to a small number of items were set to correct for 1-10% of test takers. Item difficulty, item discrimination, item fit, and local dependence were computed using an IRT 2PL model. Then changes in these item properties from the non-compromised to the compromised dataset were scrutinized for their sensitivity to item preknowledge. A decline in the discrimination parameter compared with previous test versions and an increase in local item dependence turned out to be the most sensitive indicators of item preknowledge. A multiplicative combination of shifts in item discrimination, item difficulty, and local item dependence detected item preknowledge with a sensitivity of 1.0 and a specificity of .95 if 11 of 80 items were preknown to 10% of the test takers. Cheating groups smaller than 5% of the test takers were not detected reliably. In the discussion, we outline an effective search for items affected by cheating, which would enable faculty staff without IRT knowledge to detect compromised items and exclude them from scoring.

U2 - 10.1371/journal.pone.0167545

DO - 10.1371/journal.pone.0167545

M3 - SCORING: Journal article

C2 - 27907190

VL - 11

SP - e0167545

JO - PLOS ONE

JF - PLOS ONE

SN - 1932-6203

IS - 12

ER -