Toward Automated Classroom Observation: Multimodal Machine Learning to Estimate CLASS Positive Climate and Negative Climate

  title={Toward Automated Classroom Observation: Multimodal Machine Learning to Estimate CLASS Positive Climate and Negative Climate},
  author={Anand Ramakrishnan and Brian Zylich and Erin Ottmar and Jennifer LoCasale-Crouch and Jacob Whitehill},
In this work we present a multi-modal machine learning-based system, which we call ACORN, to analyze videos of school classrooms for the Positive Climate (PC) and Negative Climate (NC) dimensions of the CLASS observation protocol that is widely used in educational research. ACORN uses convolutional neural networks to analyze spectral audio features, the faces of teachers and students, and the pixels of each image frame, and then integrates this information over time using Temporal Convolutional… Expand
1 Citations

Figures and Tables from this paper


Toward Automated Classroom Observation: Predicting Positive and Negative Climate
This work represents the first automated system designed to detect specific dimensions of the Classroom Assessment Scoring System, and constructed automated detectors of positive and negative classroom climate with accuracy of 0.40 and 0.51, respectively. Expand
Inferring the Climate in Classrooms from Audio and Video Recordings: A Machine Learning Approach
  • Anusha James, M. Kashyap, +4 authors J. Dauwels
  • Computer Science
  • 2018 IEEE International Conference on Teaching, Assessment, and Learning for Engineering (TALE)
  • 2018
A novel system for automatic assessment of classroom climate, based on speech, behavioral cues and video features by applying machine learning techniques is proposed, which can empower education systems to continuously review and improve teaching strategies thus promoting smart classroom in the future. Expand
Deep Learning for Classroom Activity Detection from Audio
This work introduces a set of deep learning classifiers for automatic activity annotation, evaluating them on a collection of classroom recordings, and shows that their estimates of how much classroom time spent per task are better correlated with actual time spent than existing systems. Expand
Automated Classification of Classroom Climate by Audio Analysis
Speech processing algorithms that detect speakers and social behavior from audio recordings in classrooms and extract non-verbal speech cues and low-level audio features from speech segments and train classifiers based on those cues indicate the potential of predicting classroom climate automatically fromaudio recordings. Expand
Noise-Robust Key-Phrase Detectors for Automated Classroom Feedback
  • Brian Zylich, J. Whitehill
  • Computer Science
  • ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)
  • 2020
This work investigates how to train automatic speech detectors of key phrases such as good job, thank you, please, and you’re welcome, on a modest-sized but highly-tailored dataset of classroom speech. Expand
Multi-sensor modeling of teacher instructional segments in live classrooms
A multi-sensor approach led to an average 8% improvement over the state of the art approach that only analyzed teacher audio, and for Supervised Seatwork, Small Group Work, and Lecture segments, the classroom model outperformed both the teacher and fusion models. Expand
Automatic Detection of Mind Wandering from Video in the Lab and in the Classroom
We report two studies that used facial features to automatically detect mind wandering, a ubiquitous phenomenon whereby attention drifts from the current task to unrelated thoughts. In a laboratoryExpand
Automatic Detection of Learning-Centered Affective States in the Wild
Computer vision and machine learning techniques were used to detect students' affect as they used an educational game designed to teach fundamental principles of Newtonian physics, resulting in a five-way overall classification of affect. Expand
Automatically Measuring Question Authenticity in Real-World Classrooms
This work set out to use automatic speech recognition, natural language processing, and machine learning to train computers to detect authentic questions in real-world classrooms automatically and provide a valuable complement to human coding in research efforts. Expand
Harnessing Label Uncertainty to Improve Modeling: An Application to Student Engagement Recognition
  • Arkar Min Aung, J. Whitehill
  • Computer Science
  • 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018)
  • 2018
This paper explores how harnessing the full probability distribution of each label, rather than just a scalar summary statistic, can yield better recognition accuracy when training automated detectors and suggests that training on soft labels can deliver engagement detectors that fit the data stat. Expand