Exploring the Relationship Between Raters’ Personality Traits and Rating Severity in Writing Assessment of Chinese as a Second Language: A Pilot Study

Abstract

Subjective rating is a widely adopted practice in human assessment and measurement. Without a careful control of severity among raters in grading, both the validity and reliability of a subjective assessment will be subject to skepticism. Among many variables which may potentially impact raters’ severity, raters’ personality in recent years has caused concerns for many researchers. However, so far there is no study investigating such a potential relationship in the context of testing and assessing Chinese as a second language. To explore the relationship between raters’ personality traits and rating severity in writing assessment of Chinese as a foreign language, master’s degree students who majored in teaching of Chinese as a foreign language (n = 28) were asked to rate a writing test of the New HSK, a large-scale standardized test of Chinese proficiency promoted by the Chinese government. Valid data for FACETS analysis were the 28 raters’ independent ratings of 77 examinees’ 154 essays (2 topics) from the test. Each rater also completed NEO-PI-R, a fairly reliable personality inventory. Demographic variables such as age, gender, educational background, and rating experience, which may affect raters’ severity in essay grading, were strictly controlled. The results of this pilot study showed individual differences in raters’ severity and significant associations between raters’ personality traits and their rating behavior in terms of severity. Findings of this study are expected to enrich our understanding of subjective rating behaviors of human assessment, especially in the field of testing and assessing Chinese as a second language.