A Learning Analytics Methodology for Student Success Efficacy Studies

migrationflowvizPI Rich Levine
Co-PIs Professors Juanjuan Fan (Statistics) and Bernie Dodge (Learning Design and Technology)

Collaborators ASIR Director Jeanne Stronach and ITS Director James Frazee

Supported through the NSF BIGDATA program and Education and Human Resources Directorate Award for just over $1 million dollars for 3 years, 2016-2019

How can instructors, administrators, and education researchers take advantage of rich institutional student information system (SIS), learning management system (LMS), and non-LMS performance data to characterize at-risk students that will most benefit from pedagogical innovations and intervention strategies?  An SDSU interdisciplinary team from statistics and data science, institutional research, and instructional technology have secured NSF funding to develop a learning analytics methodology to automate the tasks of data collection and processing, data visualizations and summaries, data analysis, and scientific reporting in student success efficacy studies.  As part of this development, the concept of individualized treatment effects will be introduced as a method to assess the effectiveness of interventions and/or instructional regimes and provide personalized feedback to students.

The research effort is under the collaborative umbrella of the SDSU Analytic Studies and Institutional Research (ASIR) Statistical Modeling Group founded by ASIR Director Jeanne Stronach and ASIR Faculty Advisor and Professor of Statistics Rich Levine.  In addition to Levine and Stronach, the research team consists of Professor of Statistics Juanjuan Fan, Professor of Learning Design and Technology Bernie Dodge, and Instructional Technology Services (ITS) Director James Frazee.  The grant, entitled “Acting on Actionable Intelligence: A Learning Analytics Methodology for Student Success Efficacy Studies” is supported through the NSF BIGDATA program and Education and Human Resources Directorate.  The award is for just over $1 million dollars for 3 years, 2016-2019.

The project entails developing a new random forest of interaction trees machine learning method that allows investigation of treatment effects for individuals and for subgroups (e.g., success of a pedagogical intervention for each student or in specific student subgroups). The approach overcomes selection biases from non-randomized treatment assignment in quasi-experimental study designs. Novel learning analytics tools will be developed to automate the tasks of collecting, cleaning, and merging SIS and LMS data, managing, storing, and easily accessing databases for student success studies, and creating student success dashboards to report findings to key stakeholders. These dashboards will allow administrators to assess intervention strategies and allocate resources accordingly, instructors to design and optimize pedagogical reforms with an eye on success for at-risk students, and educational researchers to evaluate and refine intervention strategies across disciplines.

Application of the student success efficacy study learning analytics framework developed will be aimed at addressing questions of national importance (particularly in STEM education): To what extent do pedagogical innovations and intervention strategies ameliorate the reasons that students typically leave a (STEM) field? How do non-traditional students, women, and members of underrepresented minority groups fare under intervention and instructional regimes? How can student success strategies be leveraged to increase persistence in (STEM) disciplines?