Resampling-based approaches for person fit assessment in cognitive diagnosis modeling

Abstract

Detection of aberrant response patterns is of primary importance in educational and psychological measurement because the discrepancy between examinee’s ability and test performance could lead to inappropriate remediation measures wasting teacher’s and student’s efforts or invalid selection decisions leading to serious consequences. Person Fit (PF) analysis principally aims to identify such response patterns. However, in cognitive diagnosis modelling, there is a dearth of research on PF literature. This study attempts to bridge this gap by investigating the viability of resampling-based approaches (Sinharay, 2016), whether they can provide more accurate classification of aberrant examinees than the traditional approach of assuming normality. The first procedure for resampling utilizes the parametric bootstrap (Efron, 1979). It fixes the attribute pattern and uses it to simulate bootstrapped response patterns assuming the item parameters are known. The second procedure, on the other hand, employs the posterior distribution of each response pattern and uses it to generate response patterns, also, on the assumption that the item parameters are known. Likelihood- and residual-based PF statistics are considered in this study using the Generalized Deterministic Inputs, Noisy 'And' gate (G-DINA; de la Torre, 2011) model. Preliminary simulation results showed that, compared to the traditional approach, the two resampling approaches have well-controlled Type I error rates for the likelihood-based statistics. It was found that there is a very small difference between the two resampling-based approaches and both of them work well with likelihood-based PF statistics. The residual-based statistics, on the other hand, perform very poorly in all three approaches.