Autonomously Generating Hints by Inferring Problem Solving Policies

@article{Piech2015AutonomouslyGH,
  title={Autonomously Generating Hints by Inferring Problem Solving Policies},
  author={Chris Piech and Mehran Sahami and Jonathan Huang and Leonidas J. Guibas},
  journal={Proceedings of the Second (2015) ACM Conference on Learning @ Scale},
  year={2015}
}
  • C. Piech, M. Sahami, L. Guibas
  • Published 14 March 2015
  • Computer Science
  • Proceedings of the Second (2015) ACM Conference on Learning @ Scale
Exploring the whole sequence of steps a student takes to produce work, and the patterns that emerge from thousands of such sequences is fertile ground for a richer understanding of learning. In this paper we autonomously generate hints for the Code.org `Hour of Code,' (which is to the best of our knowledge the largest online course to date) using historical student data. We first develop a family of algorithms that can predict the way an expert teacher would encourage a student to make forward… 
View on ACM
stanford.edu
126 Citations
Highly Influential Citations
21
Background Citations
61
Methods Citations
29
Results Citations
3

Figures and Tables from this paper

126 Citations

The Continuous Hint Factory - Providing Hints in Vast and Sparsely Populated Edit Distance Spaces
TLDR
This contribution provides a mathematical framework for edit-based hint policies and proposes a novel hint policy to provide edit hints in vast and sparsely populated state spaces and demonstrates that the Continuous Hint Factory can predict more accurately what capable students would do compared to existing prediction schemes on two learning tasks.
Data-Driven Hint Generation in Vast Solution Spaces: a Self-Improving Python Programming Tutor
TLDR
The results show that ITAP is capable of producing hints for almost any given state after being given only a single reference solution, and that it can improve its performance by collecting data over time.
Automated Prediction of Novice Programmer Performance Using Programming Trajectories
TLDR
This work defines a supervised machine learning algorithm that takes as input students’ programming trajectories and predicts whether a student will successfully complete the next exercise and indicates that the model can accurately predict the future performance of the students.
Deep Knowledge Tracing On Programming Exercises
TLDR
This work feeds embedded program submissions into a recurrent neural network and train it on the task of predicting the student's success on the subsequent programming exercise, and reliably predicts future student performance.
Automated Data-Driven Hint Generation in Intelligent Tutoring Systems for Code-Writing: On the Road of Future Research
TLDR
The goal in this paper is to review and classify analysis techniques that are requested to generate data-driven hints in ITSs for programming, and to identify the possible future directions in this research field.
Learning to Represent Student Knowledge on Programming Exercises Using Deep Learning
TLDR
This work feeds the embedded program submission sequence into a recurrent neural network and train it on two tasks of predicting the student’s future performance, and learns nuanced representations of a student's knowledge, exposes patterns about a student�'s learning behavior, and reliably predicts future student performance.
A Survey of Automated Programming Hint Generation: The HINTS Framework
TLDR
All hint techniques can be understood as a series of simpler components with similar properties, and a simple framework for describing such techniques is presented, the Hint Iteration by Narrow-down and Transformation Steps (HINTS) framework.
A Classification of Data-Driven Hint Generation Techniques for Code-Writing Intelligent Tutoring Systems
  • H. Bui
  • Computer Science, Education
  • 2017
TLDR
The goal in this paper is to review and classify analysis techniques that are requested to generate data-driven hints in ITSs for programming, and to propose several future research directions.
Generating Data-driven Hints for Open-ended Programming
TLDR
A new data-driven algorithm is presented, based on the Hint Factory, to generate hints for open-ended programming assignments, that can provide hints that successfully lead students to solutions from any state, help students achieve assignment objectives, and align with the student’s future solution.
Using Bugs in Student Code to Predict Need for Help
TLDR
This work uses information about the bugs present in student code to predict which students are more likely to abandon the puzzle or take too long in solving it, and envision that teachers could use these predictions to make decisions about whom they should help next, and how.
...
...

References

SHOWING 1-10 OF 25 REFERENCES
Toward Automatic Hint Generation for Logic Proof Tutoring Using Historical Student Data
TLDR
The feasibility of this approach to automatically generate hints for an intelligent tutor that learns is demonstrated by extracting MDPs from four semesters of student solutions in a logic proof tutor, and the probability that they will be able to generate hints at any point in a given problem is calculated.
Syntactic and Functional Variability of a Million Code Submissions in a Machine Learning MOOC
TLDR
The syntax and functional similarity of the submissions are mapped out in order to explore the variation in solutions in the first offering of Stanford's Machine Learning Massive Open-Access Online Course.
BlueFix: Using Crowd-Sourced Feedback to Support Programming Students in Error Diagnosis and Repair
TLDR
BlueFix is an online tool currently integrated into the BlueJ IDE which is designed to assist programming students with error diagnosis and repair and proposes a feedback algorithm based upon frameworks combined from the HCI and Pedagogical domains which can provide different students with dynamic levels of support based upon their compilation behaviour.
A Canonicalizing Model for Building Programming Tutors
TLDR
This work has constructed a language-independent canonicalized model for programming solutions that allows for much greater overlap across different students than a basic text model, which enables more self-sustaining hint generation methods in programming tutors.
On automated grading of programming assignments in an academic institution
Codewebs: scalable homework search for massive open online programming courses
TLDR
A method for decomposing online homework submissions into a vocabulary of "code phrases", and based on this vocabulary, a queryable index that allows for fast searches into the massive dataset of student homework submissions is designed.
Cognitive Tutor: Applied research in mathematics education
TLDR
The theoretical background of this approach to cognitive models of mathematics, which have become a basis for middle- and high-school curricula, and how embedding a well specified theory in instructional software allows us to dynamically evaluate the effectiveness of the authors' instruction at a more detailed level than was previously possible.
An effective metacognitive strategy: learning by doing and explaining with a computer-based Cognitive Tutor
On the Benefits of Seeking (and Avoiding) Help in Online Problem-Solving Environments
TLDR
It is suggested that novice learners may benefit from engaging in solution attempts before they can make sense of given assistance, as contrary to many help-seeking theories, avoiding help (and failing repeatedly) is associated with better learning.
Peer and self assessment in massive online classes
TLDR
This article reports the experiences with two iterations of the first large online class to use peer and self-assessment, and finds that rubrics that use a parallel sentence structure, unambiguous wording, and well-specified dimensions have lower variance.
...
...