Combining quantitative and qualitative measures to validate a group speaking assessment test

Abstract

All too often, the shift from a norm-referenced to criterion-reference assessment results in tests that reflect holistic and (teacher-biased) expectations of student performance without actually determining whether the criteria hold sufficient standards of validity and reliability, resulting in considerable inter-rater variance. This is particularly felt in L2 contexts where students struggle to achieve grades under criteria that are not produced in the same language (or spirit) as their L1, and where teachers from different backgrounds may interpret individual criteria according to their ideologies and beliefs. However, most validation studies only focus on quantitative matters (losing the personal, holistic focus) or focus on qualitative concerns (missing the general picture). The present study validates a criterion-referenced group tutorial discussion speaking assessment for undergraduate EAP at a leading university in Hong Kong in terms of inter-rater variance and criterion validity. I present three complementary quantitative measures (Intraclass Correlation Coefficient, Cronbach's Alpha and Exploratory Factor Analysis) which suggest a number of criteria could safely be removed from the rubric, and show that the grading of some criteria frequently overlaps that of other criteria. However, in attempting to explain the reasons behind the statistical results, qualitative interviews with test raters suggest that 1) raters bring with them their own interpretations of the rubric criteria and 2) there are specific linguistic considerations regarding individual test takers' performance and rater variance on said performance. The results have implications for improving the validity and reliability of in-house produced criterion-reference assessment rubrics, and hopefully the paper serves as a 'how-to' for language assessment practitioners.