Evaluation of text coherence for electronic essay scoring systems

  title={Evaluation of text coherence for electronic essay scoring systems},
  author={Eleni Miltsakaki and Karen Kukich},
  journal={Natural Language Engineering},
  pages={25 - 55}
Existing software systems for automated essay scoring can provide NLP researchers with opportunities to test certain theoretical hypotheses, including some derived from Centering Theory. In this study we employ the Educational Testing Service's e-rater essay scoring system to examine whether local discourse coherence, as defined by a measure of Centering Theory's Rough-Shift transitions, might be a significant contributor to the evaluation of essays. Rough-Shifts within students' paragraphs… 

Mining Domain Knowledge for Coherence Assessment of Students Proposal Drafts

A coherence analyzer is presented, which employs latent semantic analysis (LSA) to mine existing corpora to further assess new drafts and uncovering a consistent interrelationship, according to methodology authors.

Automated assessment of English-learner writing

This thesis argues that an approach which does not rely on task-dependent components or data, and directly assesses learner English, can produce results as good as promptspecific models, and presents a user interface that visualises and uncovers the ‘marking criteria’ represented in AA models.

Automated Essay Scoring for Swedish

We present the first system developed for automated grading of high school essays written in Swedish. The system uses standard text quality indicators and is able to compare vocabulary and grammar to

Scoring for Swedish

We present the first system developed for automated grading of high school essays written in Swedish. The system uses standard text quality indicators and is able to compare vocabulary and grammar to

Modeling coherence in ESOL learner texts

This work presents the first systematic analysis of several methods for assessing coherence under the framework of automated assessment (AA) of learner free-text responses, and examines the predictive power of different coherence models by measuring the effect on performance when combined with an AA system that achieves competitive results.

Modeling Organization in Student Essays

A new annotated corpus is presented and heuristic-based and learning-based approaches to scoring essays along the organization dimension are proposed, utilizing techniques that involve sequence alignment, alignment kernels, and string kernels.

Automated Detection of Local Coherence in Short Argumentative Essays Based on Centering Theory

An automated method for assessing local coherence in short argumentative essays is described and it is suggested that focusing on nominal subject for detecting candidate concepts for a discourse segment's central concept is sufficient, which confirms previous findings.

Modeling Thesis Clarity in Student Essays

A new annotated corpus is presented and a learning- based approach to scoring essays along the thesis clarity dimension is proposed and a second learning-based approach to identifying what kinds of errors an essay has that may lower its thesis clarity score is proposed.

Modelling text meta-properties in automated text scoring for non-native English writing

  • Meng Zhang
  • Computer Science
  • 2019
This thesis showed that explicitly modelling these properties via frustratingly easy domain adaptation (FEDA) can positively affect model performance in some cases and showed that modelling multiple intra-exam properties together is better than modelling any single property individually or no property in four out of five test sets.

Give Me More Feedback II: Annotating Thesis Strength and Related Attributes in Student Essays

A scoring rubric is designed for scoring a core, yet unexplored dimension of persuasive essay quality, thesis strength, and annotate a corpus of essays with thesis strength scores to provide further feedback on why her essay receives a particular thesis strength score.



Beyond Automated Essay Scoring

Using computers to increase the understanding of the textual features and cognitive skills involved in creating and comprehending written text will have clear benefits and help develop more effective instructional materials for improving reading, writing, and other human communication abilities.

The Computer Moves into Essay Grading: Updating the Ancient Test.

When it comes to reading essays and rating them, computers have not played much of a role to date. But that may change, according to Mr. Page and Ms. Petersen, because - for the first time ever - a

Functional Centering - Grounding Referential Coherence in Information Structure

A revision of the principles guiding the ordering of discourse entities in the forward-looking center list within the centering model is proposed, claiming that grammatical role criteria should be replaced by criteria that reflect the functional information structure of the utterances.

Discourse Segmentation by Human and Automated Means

The first part of this paper presents a method for empirically validating multitutterance units referred to as discourse segments, and reports highly significant results of segmentations performed by naive subjects, where a commonsense notion of speaker intention is the segmentation criterion.

The Measurement of Textual Coherence with Latent Semantic Analysis.

The approach for predicting coherence through reanalyzing sets of texts from 2 studies that manipulated the coherence of texts and assessed readers’ comprehension indicates that the method is able to predict the effect of text coherence on comprehension and is more effective than simple term‐term overlap measures.

Specifying the Parameters of Centering Theory: a Corpus-Based Evaluation using Text from Application-Oriented Domains

The main result is that trying to reduce the number of utterances without a backward-looking center results in an increased number of cases in which some discourse entity, but not the CB, gets pronominalized, and viceversa.

An introduction to latent semantic analysis

The adequacy of LSA's reflection of human knowledge has been established in a variety of ways, for example, its scores overlap those of humans on standard vocabulary and subject matter tests; it mimics human word sorting and category judgments; it simulates word‐word and passage‐word lexical priming data.

A Corpus-based Investigation of Definite Description Use

Questions are raised concerning the starategy of evaluating systems for definite description interpretation by comparing their results with a standardized annotation, and the great number of discourse-new definites and the presence of definites that did not seem to require a complete disambiguation.

Centering, Anaphora Resolution, and Discourse Structure

This chapter argues that the restriction of centering to operating within a discourse segment should be abandoned in order to integrate centering with a model of global discourse structure and shows how the cache model is easily integrated with the centering algorithm, and several types of data from naturally occurring discourses that support the proposed integrated model.

Japanese Discourse and the Process of Centering

It is shown that CENTERING provides constraints on when a ZERO can be interpreted as the ZERO TOPIC, and it is argued that while discourse interpretation is an inferential process, syntactic cues constrain this process.