From Lab to Production: Lessons Learnt and Real-Life Challenges of an Early Student-Dropout Prevention System

Abstract

This paper presents the work done to support student dropout risk prevention in a real online e-learning environment: A Spanish distance university with thousands of undergraduate students. The main goal is to prevent students from abandoning the university by means of retention actions focused on the most at-risk students, trying to maximize the effectiveness of institutional efforts in this direction. With this purpose, we generated predictive models based on the C5.0 algorithm using data from more than 11,000 students collected along five years. Then we developed SPA, an early warning system that uses these models to generate static early dropout-risk predictions and dynamic periodically updated ones. It also supports the recording of the resulting retention-oriented interventions for further analysis. SPA is in production since 2017 and is currently in its fourth semester of continuous use. It has calculated more than 117,000 risk scores to predict the dropout risk of more than 5,700 students. About 13,000 retention actions have been recorded. The white-box predictive models used in production provided reasonably good results, very close to those obtained in the laboratory. On the way from research to production, we faced several challenges that needed to be effectively addressed in order to be successful. In this paper, we share the challenges faced and the lessons learnt during this process. We hope this helps those who wish to cross the road from predictive modelling with potential value to the exploitation of complete dropout prevention systems that provide sustained value in real production scenarios