Developing algorithms to optimize recommender system for self-regulated learning promotion in online contexts : a design-based study

Abstract

Enhancing students’ self-regulated learning (SRL) requires an accurate assessment of students' SRL abilities to deliver personalized interventions effectively. Recommender systems have garnered significant attention due to their ability to balance external regulation with learner autonomy. However, deploying such systems to enhance SRL poses challenges, particularly in identifying the most appropriate interventions for individual students. Certain algorithms, especially those designed to analyze and evaluate students' SRL levels, show promise in addressing this difficulty. Accurate assessment of students' SRL capabilities is crucial for providing personalized instruction. This research conducted a design-based study in three cycles to optimize assessment algorithms within a recommender system aimed at supporting students’ SRL. The goal was to develop more effective algorithms that enhance both the accuracy of SRL assessments and the practical effectiveness of fostering SRL development. By controlling for variables such as instructional environment and content, this study focused specifically on algorithm optimization to improve SRL outcomes. The effectiveness of SRL promotion was evaluated from three perspectives: students' behavioral engagement, SRL skill development, and learning performance improvement. This three-cycle DBR study involved 107 university students (Cycle 1: n=23, Cycle 2: n=26, Cycle 3: n=58) participating in an online English essay writing course. In this three-cycle DBR study, three distinct algorithmic approaches were iteratively developed and evaluated: (1) a rule-based approach grounded in SRL theory, (2) a fully-supervised machine learning approach utilizing XGBoost algorithms, and (3) a hybrid approach that integrated rule-based approaches with semi-supervised machine learning while incorporating human expertise. The recommender system using this hybrid approach demonstrated superior SRL outcomes compared to systems relying solely on rule-based or supervised machine learning. Results demonstrated that the hybrid approach achieved the highest SRL skill gains (d=1.62), engagement levels (365.16 vs. 239.96 indicators), and improved prediction accuracy (R²=0.60-0.85 across subprocesses). This study offers insights and recommendations for the future development of SRL promotion interventions.