Problems Before Solutions: Automated Problem Clarification at Scale

  title={Problems Before Solutions: Automated Problem Clarification at Scale},
  author={Soumya Sankar Basu and Albert Wu and Brian Hou and John DeNero},
  journal={Proceedings of the Second (2015) ACM Conference on Learning @ Scale},
  • S. Basu, Albert Wu, John DeNero
  • Published 14 March 2015
  • Education, Psychology
  • Proceedings of the Second (2015) ACM Conference on Learning @ Scale
Automatic assessment reduces the need for individual feedback in massive courses, but often focuses only on scoring solutions, rather than assessing whether students correctly understand problems. We present an enriched approach to automatic assessment that explicitly assists students in understanding the detailed specification of technical problems that they are asked to solve, in addition to evaluating their solutions. Students are given a suite of solution test cases, but they must first… 

Figures from this paper

Answering the Correct Question
An randomized A/B test with 831 CS1 students using an online submission system is reported on, hypothesized that this would decrease the time and attempts required to correctly write the code because students in the treatment group would not waste time on an incorrectly-conceived problem.
Giving hints is complicated: understanding the challenges of an automated hint system based on frequent wrong answers
This work set out to deliver formative feedback to students on code-tracing, constructed-response assessments by building a student error model using insights gained from inspecting the assessment's frequent wrong answers, and compared two different kinds of hints: reteaching and knowledge integration.
High-Coverage Hint Generation for Massive Courses: Do Automated Hints Help CS1 Students?
A robust hint generation system that extends the coverage of the mutation-based approach using two complementary techniques and shows that hints contributed to students' progress while still encouraging the students to solve problems by themselves.
Identifying Student Misunderstandings using Constructed Responses
This work proposes a technique for using each student's constructed responses in order to identify a subset of their stable conceptual misunderstandings, which substantially outperforms a baseline that does not take advantage of co-occurrence statistics, proves robust to different course sizes, and generalizes effectively across student cohorts.
High Coverage Hint Generation for Massive Courses by Sumukh Sridhara Research Project
A robust hint generation system that extends the coverage of the mutation-based approach using two complementary techniques and shows that hints contributed to students’ progress while still encouraging the students to solve problems by themselves is described.
An exploration of learning tool log data in CS1: how to better understand student behaviour and learning
BitFit, an ungraded practice programming tool built to provide students with a pressure-free environment to practice and build confidence working through weekly course material, is introduced, providing insight into the factors contributing to behavioural change in an introductory programming course.
The Challenges of Evolving Technical Courses at Scale: Four Case Studies of Updating Large Data Science Courses
Four case studies from the authors' teaching experiences highlight major challenges in maintaining and updating technical courses and design ideas for instructor-oriented tools that can reduce the logistical complexities of teaching at scale, thus letting instructors focus on the substance of teaching rather than on mundane logistics.
Providing personalized learning guidance in MOOCs by multi-source data analysis
This paper provides personalized learning guidance for MOOC students based on multi-source data analysis, and proposes two structured model to evaluate student knowledge states by their quiz submissions.
Students, systems, and interactions: synthesizing the first four years of learning@scale and charting the future
Future research directions related to topics such as going beyond the MOOC hype cycle, axes of scale for systems, more immersive course experiences, learning on mobile devices, diversity in student personas, students as co-creators, and fostering better social connections amongst students are charted.
An Effective Framework for Automatically Generating and Ranking Topics in MOOC Videos
A novel framework that automatically generates and ranks the topics within the upcoming chapter through a particular PageRank method, which also leverages structural information of lectures to show students the emphasis of lectures before their learning.


Review of recent systems for automatic assessment of programming assignments
It is stated that too many new systems are developed, but also acknowledge the current reasons for the phenomenon, and encourages opening up the existing systems and joining efforts on developing those further.
Divide and correct: using clusters to grade short answers at scale
A cluster-based interface is proposed that allows teachers to read, grade, and provide feedback on large groups of answers at once and is found to allow teachers to grade substantially faster, to give more feedback to students, and to develop a high-level view of students' understanding and misconceptions.
The Critical Importance of Retrieval for Learning
The results demonstrate the critical role of retrieval practice in consolidating learning and show that even university students seem unaware of this fact.
Hint systems may negatively impact performance in educational games
The results suggest that traditional hint systems may not translate well into the educational game environment, highlighting the importance of studying student behavior to understand the impact of new interactive learning technologies.
How to design programs: an introduction to programming and computing
This introduction to programming places computer science in the core of a liberal arts education and focuses on the program design process, which fosters a variety of skills -- critical reading, analytical thinking, creative synthesis, and attention to detail -- that are important for everyone, not just future computer programmers.
Nifty assignments
The presenter will introduce their assignment, give a quick demo, and describe its niche in the curriculum and its strengths and weaknesses, and introduce each assignment.
Assessment in higher education: student learning, teaching, programmes and institutions / John Heywood
Acknowledgements. Dedication. Foreword, Thomas Angelo. 1. Assessment in Higher Education: Clarifying Issues and Terms. 2. Toward Multiple Strategy Assessment. 3. Assessment and Accountability:
Blogs and forums as communication and learning tools in a MOOC
This paper presents the findings of research carried out into the use of blogs and forums as communication and learning tools in Connectivism and Connective Knowledge (CCK08), a Massive Open Online
Assessment in higher education
Assessment in higher education - an introduction en-route to a degree grades and grading temperament and testing defining the task of assessment - implications for teaching finding objectives for
Assessment and ‘third generation’ distance education
Course designers are seeking to build constructivist pedagogy into the learning process so that students will perceive the use of computer mediated communication as essential for the achievability of their learning.