Impact of Question Difficulty on Engagement and Learning

  title={Impact of Question Difficulty on Engagement and Learning},
  author={Jan Papousek and V{\'i}t Stanislav and Radek Pel{\'a}nek},
  booktitle={International Conference on Intelligent Tutoring Systems},
We study the impact of question difficulty on learners' engagement and learning using an experiment with an open online educational system for adaptive practice of geography. The experiment shows that easy questions are better for short term engagement, whereas difficult questions are better for long term engagement and learning. These results stress the necessity of careful formalization of goals and optimization criteria of open online education systems. We also present disaggregation of… 

Evaluation of Learners' Adjustment of Question Difficulty in Adaptive Practice of Facts

A version of an adaptive educational system for geography practice that allows learners to adjust difficulty of questions is implemented using a randomized control experiment to provide insight into how to tune question difficulty automatically.

Should We Give Learners Control Over Item Difficulty?

A version of an adaptive educational system for geography practice that allows learners to adjust difficulty of questions is implemented using a randomized control experiment to provide insight into how to tune question difficulty automatically.

Learning analytics challenges: trade-offs, methodology, scalability

It is argued that learning analytics research should reflect this nature of the education domain and focus on less clearly defined, but practically essential issues.

Elo-based learner modeling for the adaptive practice of facts

The predictive accuracy of different learner models is compared, insights gained from learner modeling are discussed, as well as the impact different variants of the system have on learners’ engagement and learning.

Assessing Student's Dynamic Knowledge State by Exploring the Question Difficulty Effect

The DIfficulty Matching Knowledge Tracing (DIMKT) model is proposed and it is demonstrated that DIMKT outperforms state-of-the-art KT models and shows superior interpretability by exploring the question difficulty effect when making predictions.

Predicting Students’ Disengaged Behaviors in an Online Meaning-Generation Task

Data-driven methods were used to address two aspects of this problem: identification of predictive features at the single-trial level, and selection of accurate and robust models for representing and modeling temporal patterns of student engagement in an ITS, especially those related to language learning.

Towards the Application of Calibrated Transformers to the Unsupervised Estimation of Question Difficulty from Text

The uncertainty of calibrated question answering models are used as a proxy of human-perceived difficulty and suggest that model uncertainty could be successfully leveraged to perform QDE from text, reducing both costs and elapsed time.

Personalized task difficulty adaptation based on reinforcement learning

The proposed algorithms were combined and applied to a real-world application consisting of an online visual-spatial memory recall game and were shown to outperform the traditional rule-based adaptation approach in adapting to the slow players while achieving comparable performance in adaptingto the fast players.

Bootstrapped Policy Gradient for Difficulty Adaptation in Intelligent Tutoring Systems

A bootstrapped policy gradient (BPG) framework, which can incorporate prior knowledge into policy gradient to enhance sample efficiency and achieves fast and unbiased convergence both in theory and in practice.



Impact of Adaptive Educational System Behaviour on Student Motivation

This work tries to connect research on student modeling and student motivation, particularly on the relation between task difficulty and engagement, with widely used adaptive practice system for geography learning.

Evaluation of an adaptive practice system for learning geography facts

This case study uses an online system for adaptive practice of geography to highlight methodological issues with evaluation of open online learning systems, particularly attrition bias, and proposes to use randomized reference questions to facilitate evaluation of learning.

Adaptive Practice of Facts in Domains with Varied Prior Knowledge

It is argued that variants of the Elo rating systems and Performance factor analysis are suitable for this kind of educational system, as they provide good accuracy and at the same time are easy to apply in an online system.

Impact of data collection on interpretation and evaluation of student models

Two ways how data collection in educational systems can influence results: mastery attrition bias and adaptive choice of items are discussed.

Evaluating and improving adaptive educational systems with learning curves

This work describes the use of learning curves for evaluating personalised educational systems and outlines some of the potential pitfalls and how they may be overcome, and demonstrates the successful use of performance curves in formative studies of adaptive educational systems.

Optimizing challenge in an educational game using large-scale design experiments

This research manipulated the design of an online math game to investigate the effect of challenge on player motivation and learning, and found that the most engaging design conditions produced the slowest rates of learning.

Towards automatic experimentation of educational knowledge

A general automatic experimentation and hypothesis generation framework that utilizes a large set of users to explore the effects of different parts of an intervention parameter space on any objective function, and an implementation in the domain of fractions and numberlines is presented.

Revealing the Learning in Learning Curves

It is shown that disaggregating a skill’s aggregate learning curve into separate learning curves for different student subpopulations can reveal learning: 70% of the skills that did not show learning and were identified as candidates for improvement did show learning when disaggregated.

Student Models for Prior Knowledge Estimation

This work describes several models for prior knowledge estimation – the Elo rating system, its Bayesian extension, a hierarchical model, and a networked model (multivariate Elo) and evaluates their performance on data from application for learning geography, which is a typical case with highly varied prior knowledge.