Inter-rater reliability investigation for reading and oral comprehension tasks

Abstract

Inter-rater reliability is the degree to which different examiners employ the same criteria for evaluating test results. The present study aims to investigate the inter-rater reliability for two tasks, one of reading comprehension and another for oral comprehension. The Reading Comprehension Task consists of 15 texts divided into two blocks (Form A and Form B), followed by orally answered openended questions. The Oral Comprehension Task (OCT) contains eight open-ended questions for a narrative (orally answered and recorded). The sample consisted of 218 children from 2nd to 5th year of elementary school in Sao Paulo (73 in Form A; 56 in Form B; 89 in OCT) and nine examiners (three for each form). Fleiss' Kappa was used to obtain the reliability index. Most items had adequate levels of agreement, which evidenced the consistency of the correction system.