Generalizing indices of classification accuracy for cognitively diagnostic assessments

Abstract

An index of classification accuracy was recently proposed that, in addition to being relatively straightforward and fast to compute, estimates accuracy conditional on the latent class rather than marginalized to the test level. This can inform the practitioner of the effectiveness of the assessment in correctly classifying specific latent classes of interest. Additionally, weighting and summing over the latent classes returns an estimate of the test-level classification accuracy for any attribute distribution of interest. This is of particular significance because a key component of the validity argument is understanding how the classification accuracy generalizes across other examinee populations. A simulation study was designed to evaluate how well the index predicted test-level accuracy for samples drawn from different attribute distributions. First, the CDM was fitted to item responses from examinees drawn from a uniform attribute distribution. The index was computed, and then weighted according to the latent-class proportions of the higher-order attribute distribution, and summed. This value was compared to the empirical classification accuracy of the CDM under the higher-order distribution. The reverse scenario was also considered. Additionally, factors manipulated include sample size, test length, and item quality. Results suggested that using the proposed index to predict classification accuracy for a different attribute distribution led to estimates close to the empirical values under all but the least favorable test conditions. In 38 out of 48 total conditions, the proposed index predicted the classification accuracy within 0.03 of the empirical value. Detailed findings will be presented.