Evaluating the performance of the uncorrected and corrected mean reliability alpha for range restriction in meta-analysis: A Monte Carlo study

Abstract

Objectives: The procedure for conducting a meta-analysis of the reliability alpha has recently been proposed by Rodriguez and Maeda (2006). Since then, a large number of studies have used this procedure (e.g. Warne, 2011), so as to provide a mean reliability estimate in a research domain. Despite its recent use, this estimate is usually smaller than its true value, when the sample (e.g. college, clinical) in each single study is subject to range restriction. This study, therefore, seeks to evaluate the performance of the uncorrected and bias-corrected mean alphas based on a Monte Carlo study. Design: Six factors were evaluated: sample size, selection ratio, selection procedure, true alpha, and numbers of items and studies.

Methods: Simulated examinees’ scores were generated through a Monte Carlo study – a widely used strategy for evaluating the robustness of a data-analytic method across replications. These examinees would then undergo a selection process; this produced a restricted sample in each singe study. Repeating this process for multiple studies in a research domain, the uncorrected and bias-adjusted mean alphas were estimated. Consequently, their performances were evaluated systematically across the simulation conditions.

Results: The results showed that the uncorrected mean alpha was undesirable, with a maximum bias of 99.7 per cent below its true value. By contrast, the bias-corrected mean alpha was generally accurate, with biases ranging from –9.4 per cent to 7.3 per cent.

Conclusions: The bias-corrected mean alpha was accurate, thereby providing a trustworthy method for reliability evaluation. This will likely change the conceptions of many mean alphas reported in recent meta-analyses.