Language assessment raters in the call centre industry: Interfaces between language and subject matter experts

Abstract

Millions of customer services representatives are assessed each year by subject matter experts (e.g., recruiters, team leaders) in Asian contact centres to ensure good spoken communication skills when serving customers on the phones. In other workplace contexts, language experts are employed to do this work but in Asian contact centres, a successful transfer of language expert assessment knowledge and skills to subject matter experts is preferred for practical and cost reasons. To date however, no studies have been carried out to demonstrate that subject matter experts can do this language assessment in a reliable way. This study explores how a linguistically-informed spoken assessment rubric, called the Business Performance Assessment Scale (BUPLAS), is used by subject matter experts to assess the English communication skills of customer service representatives employed in an India-based contact centre. After a week-long training programme on how to use BUPLAS, the results revealed that the subject matter experts had high intra-rater and inter-rater reliability when they scored real calls. They also understood, interpreted, and ‘indigenously scored’ the linguistically-informed criteria. The implications of these results in deferring to subject matter experts to conduct language assessments in this workplace are discussed.