Using Log Data to Evaluate MOOC Engagement and Inform Instructional Design

Abstract

Traditional educational studies verify the performance of courses through questionnaires, interviews and observations, which can be an arduous task for researchers. It is easier to verify the effectiveness of online courses as all the interactions between students and the courseware are recorded. However, the utilization of these activity data is lack of theoretical framework. In this paper, we propose to utilize learning interaction theory and web analytics knowledge to evaluate MOOC engagement and inform instructional design. This framework is composed of learner-interface, learner-content and learner-community interaction. 15 indicators derived from web analytics are proposed to help teachers better understand the engagement level of their courses in three interaction dimensions. To illustrate how the above analysis can facilitate teaching in practice, we used log data of 10 MOOCs owned by The University of Hong Kong on edX. Results and corresponding insights are offered. 10 experts are invited to evaluate the proposed framework. Most of them have showed positive attitudes. In the future, we will cooperate with MOOC designers and verify whether this framework can help them teaching and improve MOOC engagement.