Perceptual evaluation of voice quality change using paired comparison method

Abstract

This study investigated the inter-rater and intra-rater reliability of ratings in evaluating voice quality change of a patient using a paired comparison method with a seven-point equal appearing interval scale. Thirty-one naïve listeners, who had no prior perceptual training, and three expert listeners, who had at least five years experience in perceptual voice evaluation, completed a perceptual rating task using the paired comparison method. Results showed that expert listeners achieved a moderate inter-rater reliability (ICC = 0.529) and naïve listeners achieved a fair inter-rater reliability (ICC = 0.347). Intra-rater reliability for expert listeners was significantly higher than naïve listeners (U = 10.0, z = - 2.22, p < .05, r = − 0.380). The findings indicated that paired comparison method could be a reliable method for expert listeners in detecting perceptual voice quality change. However, naïve listeners who had no previous perceptual training, may not give as reliable ratings using paired comparison method.