Effectiveness of Chatbots in Improving Language Learning: A Meta-Analysis of Comparative Studies

Abstract

The development of artificial intelligence has presented new opportunities and challenges for language education. Using artificial intelligence techniques such as automatic speech recognition and generative artificial intelligence, conversational chatbots have been integrated into language learning and teaching. However, findings of the impact of chatbots on second and/or foreign language learning (L2) are mixed. Therefore, this research conducted a meta-analysis to synthesize the existing research. Thirty-one studies with 41 effect sizes were included in this meta-analysis. It was found that chatbots have a medium effect on L2 learning (g = 0.608, Z = 6.77, p < 0.001). Moderating analysis further showed that the effects were moderated by the mobility of access to chatbot, modality of the chatbot, whether or not generative AI technology was involved, and the nature of comparison groups in the research design. The findings suggest that chatbots in general have positive effects on L2 learning. The results suggest that to enhance the advantages of chatbots, their accessibility, modality and adaptability need to be considered. Furthermore, a more refined perspective is needed when discussing the benefits of the use of chatbots in language learning process.