Impact of item-writing flaws in multiple-choice questions on student achievement in high-stakes assessments

Abstract

Background: Multiple-choice questions (MCQs) are a common assessment method in health-science disciplines. This study reports the impact of item-writing flaws (IWFs) in MCQs on student achievement in high-stakes assessments. Summary of work: MCQ items on 10 summative test papers administered to undergraduate nursing students were reviewed for 15 commonly occurring IWFs. For each paper, two scales were computed: a total scale representing the test as delivered and a standard scale reflecting a theoretical test with only unflawed items. The pass-fail results and students with >80% outcomes were recorded. Summary of results: The proportion of flawed items on the 10 test papers ranged from 28-75%; 47.3% of all items were flawed. Fewer examinees passed the standard scale than the total scale (748 vs. 779; 90.6% vs. 94.3%). Conversely, the proportion of examinees obtaining a score of ≥80% was higher on the standard scale than the total scale (173 vs. 120; 20.9% vs. 14.5%). Conclusions: Flawed MCQ items were common in high-stakes nursing assessments. In contrast to previous research however, flawed items benefitted the test scores of borderline students and test scores of high-achieving students were adversely affected. Take-home message: Greater effort must be placed on improving the quality of MCQs used in assessments. It is important that teachers are provided with adequate training in writing high-quality test items and that all tests and examinations are subjected to adequate review both prior to and after administration.