Automated feedback generation for introductory programming assignments

  title={Automated feedback generation for introductory programming assignments},
  author={Rishabh Singh and Sumit Gulwani and Armando Solar-Lezama},
  journal={Proceedings of the 34th ACM SIGPLAN Conference on Programming Language Design and Implementation},
We present a new method for automatically providing feedback for introductory programming problems. In order to use this method, we need a reference implementation of the assignment, and an error model consisting of potential corrections to errors that students might make. Using this information, the system automatically derives minimal corrections to student's incorrect solutions, providing them with a measure of exactly how incorrect a given solution was, as well as feedback about what they… 

Figures and Tables from this paper

Feedback generation for performance problems in introductory programming assignments

A light-weight programming language extension that allows a teacher to define an algorithmic strategy by specifying certain key values that should occur during the execution of an implementation that automatically produces feedback on performance aspects in introductory programming assignments is proposed.

Verifix: Verified Repair of Programming Assignments

The results show that generating verified repair without sacrificing the overall repair rate is possible, and the implementation, Verifix, is shown to outperform Clara, a state-of-the-art tool, in terms of repair rate.

Automatic diagnosis and correction of logical errors for functional programming assignments

A new error-correction algorithm for functional languages is presented, which combines statistical error-localization and type-directed program synthesis enhanced with components reduction and search space pruning using symbolic execution.

Automated clustering and program repair for introductory programming assignments

A novel fully automated program repair algorithm for introductory programming assignments, which enables automation and scalability, is to use the existing correct student solutions to repair the incorrect attempts.

Re-Factoring Based Program Repair Applied to Programming Assignments

This work proposes a fully automated approach for generating student program repairs in real-time by first re-factoring all available correct solutions to semantically equivalent solutions and opine that this method is applicable not only to programming assignments, and could be seen as a general-purpose program repair method that can achieve good results with just a single correct reference solution.

An Incremental Hint System For Automated Programming Assignments

An incremental hint system for programming exercises is developed and evaluated that suggests that students who needed assistance with an exercise used the hint system and found it helpful to guide them through the process of building a solution.

Automated Correction for Syntax Errors in Programming Assignments using Recurrent Neural Networks

A technique for providing feedback on syntax errors that uses Recurrent neural networks (RNNs) to model syntactically valid token sequences that can fix the error by either replacing or inserting the predicted token sequence at the error location.

InFix: Automatically Repairing Novice Program Inputs

Surprisingly, this simple approach produces high quality repairs; humans judged the output of InFix to be equally helpful and within 4% of the quality of human-generated repairs.

A feasibility study of using automated program repair for introductory programming assignments

This paper adopts a new repair policy akin to the hint generation policy employed in the existing ITSP, and admits partial repairs that address part of failing tests, which results in 84% improvement of repair rate.

Providing Meaningful Feedback for Autograding of Programming Assignments

A methodology for extending autograders to provide meaningful feedback for incorrect programs and it is found that the hints given for erroneous submissions should be helpful for 96% or more of the cases.



Automated Semantic Grading of Programs

A simple language for describing error models in terms of correction rules is introduced, and a rule-directed translation strategy is formally defined that reduces the problem of finding minimal corrections in an incorrect program to the problems of synthesizing a correct program from a sketch.

Automated Grading of DFA Constructions

This paper provides a solution to automatic grading of the standard computation-theory problem that asks a student to construct a deterministic finite automaton (DFA) from the given description of its language, and provides algorithms for transforming MOSEL descriptions into DFAs and vice-versa.

Automatic program debugging for intelligent tutoring systems

The research presented provides a methodology and implementation for the diagnosis and correction of nontrivial recursive programs by repairing induction proofs in the Boyer‐Moore logic.

Automatically Generating Algebra Problems

Computer-assisted techniques for helping with pedagogy in Algebra are proposed that can be used by teachers in making examinations where they need to test students on problems similar to what they taught in class, and by students in generating practice problems tailored to their specific needs.

A trace-based framework for analyzing and synthesizing educational progressions

This work shows how it can use execution traces as a framework for abstracting the characteristics of a given procedure and defining a partial ordering that reflects the relative difficulty of two traces and can predict user evaluation of procedural difficulty better than baseline methods.

Automatic test-based assessment of programming: A review

A number of influential automatic assessment systems are reviewed, including descriptions of the earliest systems, and some of the most recent developments are presented.

Program synthesis by sketching

Sketching is introduced, a new style of synthesis that offers a fresh approach to the synergy problem and shows that sketching is a viable approach to making synthesis practical in a general programming context.

A genetic programming approach to automated software repair

The paper describes the method, reviews earlier experiments that repaired 11 bugs in over 60,000 lines of code, reports results on new bug repairs, and describes experiments that analyze the performance and efficacy of the evolutionary components of the algorithm.

From program verification to program synthesis

The proposed technique synthesizes programs for complicated arithmetic algorithms including Strassen's matrix multiplication and Bresenham's line drawing; several sorting algorithms; and several dynamic programming algorithms using verification tools built in the VS3 project.