• Corpus ID: 49210787

Enabling End-To-End Machine Learning Replicability: A Case Study in Educational Data Mining

@article{Gardner2018EnablingEM,
  title={Enabling End-To-End Machine Learning Replicability: A Case Study in Educational Data Mining},
  author={Josh Gardner and Yuming Yang and R. Baker and Christopher A. Brooks},
  journal={ArXiv},
  year={2018},
  volume={abs/1806.05208}
}
The use of machine learning techniques has expanded in education research, driven by the rich data from digital learning environments and institutional data warehouses. However, replication of machine learned models in the domain of the learning sciences is particularly challenging due to a confluence of experimental, methodological, and data barriers. We discuss the challenges of end-to-end machine learning replication in this context, and present an open-source software toolkit, the MOOC… 
[PDF] Semantic Reader
12 Citations
Background Citations
6
Methods Citations
4

Figures from this paper

12 Citations

MORF: A Framework for Predictive Modeling and Replication At Scale With Privacy-Restricted MOOC Data
TLDR
MORF has the potential to accelerate and democratize research on its massive data repository, which currently includes over 200 MOOCs, as demonstrated by initial research conducted on the platform.
Towards Portability of Models for Predicting Students’ Final Performance in University Courses Starting from Moodle Logs
TLDR
The results obtained show that it is only feasible to directly transfer predictive models or apply them to different courses with an acceptable accuracy and without losing portability under some circumstances.
Data Mining Techniques and Machine learning Algorithms in the Multimedia System to Enhance Engineering Education
TLDR
The findings suggest that more critical choices can boost employment prospects and overall educational development by implementing the new engineering education system.
The Challenge of Reproducible ML: An Empirical Study on The Impact of Bugs
TLDR
The fundamental factors that cause non-determinism in ML systems are established and a comprehensive methodology is proposed to collect buggy versions of ML libraries and run deterministic ML experiments using ReproduceML.
Evaluating Gaming Detector Model Robustness Over Time
TLDR
This research evaluates the robustness/degradation of gaming detectors when trained on older data logs and evaluated on current data logs, and demonstrates that a classic decision tree algorithm maintained its performance while more contemporary algorithms struggled to transfer to new data, even though they exhibited better performance on unseen students in both New and Old data sets by themselves.
Generalization of Machine Learning Approaches to Identify Notifiable Conditions from a Statewide Health Information Exchange.
TLDR
Free-text laboratory data from a Health Information Exchange network is leveraged to evaluate ML generalization using Notifiable Condition Detection for public health surveillance as a use case and determined that weak generalization was influenced by variant syntactic nature of free-text datasets across each lab system.
Parallel Sentimental Analysis Based on Nectar Research Cloud and AURIN
TLDR
The system built a comprehensive structure for data harvesting, NLP, feature selection, machine learning, data mining, database, Restful style API and front-end data visualization, which can be circulated on a cloud system called Nectar research cloud, and discusses the choke point of multiple-core when dealing with the parallel computing.
Issues in the Reproducibility of Deep Learning Results
TLDR
This work uses TensorFlow as the core machine learning library for the authors' deep learning systems, and routinely employ multiple GPUs to accelerate the training process.
Does the Market of Citations Reward Reproducible Work?
TLDR
This work proposes a hierarchical Bayesian model that incorporates the citation rate over time, rather than the total number of citations after a fixed amount of time, and shows that under current evidence the answer is more likely that certain fields of study do correlate reproducible works with more citations, but other fields appear to have no relationship.
A Siren Song of Open Source Reproducibility
TLDR
It is argued that venues must take more action to advance reproducible machine learning research today and there is a lack of evidence for effective actions taken by conferences to encourage and reward reproducibility.
...
...

References

SHOWING 1-10 OF 30 REFERENCES
Replicating MOOC predictive models at scale
TLDR
This work demonstrates the importance of replication of predictive modeling research in MOOCs using large and diverse datasets, illuminates the challenges of doing so, and describes the freely available, open-source software framework to overcome barriers to replication.
The Need for Open Source Software in Machine Learning
TLDR
It is argued that the situation can be significantly improved by increasing incentives for researchers to publish their software under an open source model, and a resource of peer reviewed software accompanied by short articles would be highly valuable to both the machine learning and the general scientific community.
Reproducibility in Machine Learning-Based Studies: An Example of Text Mining
TLDR
What information about text mining studies is crucial to successful reproduction of such studies is considered, including a set of factors that affect reproducibility based on the experience of attempting to reproduce six studies proposing text mining techniques for the automation of the citation screening stage in the systematic review process.
A Data Repository for the EDM Community: The PSLC DataShop
TLDR
In recent years, educational data mining has emerged as a burgeoning new area for scientific investigation because of the increasing availability of fine-grained, extensive, and longitudinal data on student learning.
Deep Knowledge Tracing
TLDR
The utility of using Recurrent Neural Networks to model student learning and the learned model can be used for intelligent curriculum design and allows straightforward interpretation and discovery of structure in student tasks are explored.
Computing Environments for Reproducibility: Capturing the "Whole Tale"
Temporal Models for Predicting Student Dropout in Massive Open Online Courses
  • Mi Fei, D. Yeung
  • Computer Science
    2015 IEEE International Conference on Data Mining Workshop (ICDMW)
  • 2015
TLDR
Based on extensive experiments conducted on two MOOCs offered on Coursera and edX, a recurrent neural network (RNN) model with long short-term memory (LSTM) cells beats the baseline methods as well as other proposed methods by a large margin.
Student success prediction in MOOCs
TLDR
This article presents a categorization of MOOC research according to the predictors, prediction, and underlying theoretical model, and critically survey work across each category, providing data on the raw data source, feature engineering, statistical model, evaluation method, prediction architecture, and other aspects of these experiments.
OpenML: A Collaborative Science Platform
TLDR
OpenML is a novel open science platform that provides easy access to machine learning data, software and results to encourage further study and application and features a web API which is being integrated in popular machine learning tools such as Weka, KNIME, RapidMiner and R packages.
An introduction to Docker for reproducible research
TLDR
How the popular emerging technology Docker combines several areas from systems research - such as operating system virtualization, cross-platform portability, modular re-usable elements, versioning, and a 'DevOps' philosophy, to address these challenges is examined.
...
...