An Investigation of DIF in the Assessment of English Listening Comprehension

Abstract

1. Aims and key words The study employed a Rasch DIF (Differential Item Functioning) approach to investigate bias of items that test academic listening comprehension by students from different educational backgrounds, specifically, Macau and Hong Kong. The objective of the study was to (1) determine if there is evidence of DIF based on educational background and (2) gain a better understanding of the processes involved in listening comprehension and the potential applicability of Aryadoust’s (2017) Integrated Cognitive Theory of Comprehension (ICTC) to the context of academic listening in English at university level. Keywords: listening comprehension, DIF, diagnostic assessment, cognitive theory 2. Sample The study involved first-year undergraduate students (in 2016-17) from one university in Macau (1464 students) and two universities in Hong Kong (1060 students). At these universities, students from all disciplines are required to take the Diagnostic English Language Tracking Assessment (DELTA), which consists of text-based listening, reading, and grammar components, as well as discrete vocabulary items. 3. Method DELTA uses the Rasch Measurement Dichotomous Model to calibrate test items and generate diagnostic reports. For this study, the listening item data from the two groups was combined in one analysis. To identify DIF, principle component analysis was first carried out to test dimensionality (i.e., to establish that the items test one latent trait). Then uniform and non-uniform DIF analysis was conducted to determine whether DIF was due to the interaction of educational background with student ability. Potential causes of uniform and non-uniform DIF were identified by expert judgment of test developers and teachers who have taught both groups of students. 4. Results Of the 309 listening items, 118 exhibited uniform DIF and 58 of these showed non-uniform DIF. Bias was identified on 48 items either for or against high and low-ability test takers from the two groups and 10 items had lower measures than expected across the whole set. The items came from a variety of texts and targeted different listening subskills. The expert judgment interviews identified potential causes of the DIF identified, such as text topic or accent familiarity, speed or length of recording and so on. These results provide insight into the impact of educational background on the subskills of listening comprehension. 5. Conclusions The findings contribute to a better understanding of comprehension and how it is impacted by educational background, providing support for the ICTC.