Pushing on Text Readability Assessment: A Transformer Meets Handcrafted Linguistic Features

@inproceedings{Lee2021PushingOT,
  title={Pushing on Text Readability Assessment: A Transformer Meets Handcrafted Linguistic Features},
  author={Bruce W. Lee and Yoonna Jang and Jason Hyung-Jong Lee},
  booktitle={EMNLP},
  year={2021}
}
We report two essential improvements in readability assessment: 1. three novel features in advanced semantics and 2. the timely evidence that traditional ML models (e.g. Random Forest, using handcrafted features) can combine with transformers (e.g. RoBERTa) to augment model performance. First, we explore suitable transformers and traditional ML models. Then, we extract 255 handcrafted linguistic features using self-developed extraction software. Finally, we assemble those to create several… 

A Neural Pairwise Ranking Model for Readability Assessment

TLDR
This paper proposes the first neural pairwise ranking model for ARA, and shows the first results of cross-lingual, zero-shot evaluation of ARA with neural models.

Automatic Readability Assessment of German Sentences with Transformer Ensembles

TLDR
The ability of ensembles of fine-tuned GBERT and GPT-2-Wechsel models to reliably predict the readability of German sentences is studied and the dependence of prediction performance on ensemble size and composition is investigated.

Auto-Select Reading Passages in English Assessment Tests?

TLDR
A method to auto-select reading passages in English assessment tests and some key insights that can be helpful in re-lated passages are shown and the future develop-ments to improve automated reading passage selection are described.

Everybody likes short sentences - A Data Analysis for the Text Complexity DE Challenge 2022

TLDR
The modeling approach for this shared task utilizes off-the-shelf NLP tools for feature engineering and a Random Forest regression model that identified the text length, or resp.

Uniform Complexity for Text Generation

TLDR
UCTG is introduced which serves as a challenge to make existing models generate uniformly complex text with respect to inputs or prompts used and lays down potential methods and approaches which can be incorporated into the general framework of steer-ing language models towards addressing this important challenge.

Efficient Machine Translation Corpus Generation

TLDR
The effect of prioritizing with the proposed method on the resulting MT corpus quality is presented versus scheduling hypotheses randomly and proves the lifecycle of MT models by focusing the linguist effort on production samples and hypotheses, which matter most for expanding MT corpora to be used for re-training them.

Overview of the GermEval 2022 Shared Task on Text Complexity Assessment of German Text

TLDR
The GermEval 2022 shared task is designed as text regression in which participants developed models to predict complexity of pieces of text for a German learner in a range from 1 to 7.

HIIG at GermEval 2022: Best of Both Worlds Ensemble for Automatic Text Complexity Assessment

TLDR
HIIG’s best-performing model for the task of automatically determining the complexity level of a German-language sentence is a combination of a transformer model and a classic feature-based model, which achieves a mapped root square mean error of 0.446 on the test data.

Trends, Limitations and Open Challenges in Automatic Readability Assessment Research

TLDR
A brief survey of contemporary research on developing computational models for readability assessment identifies the common approaches, discusses their shortcomings, and identifies some challenges for the future.

SPADE: A Big Five-Mturk Dataset of Argumentative Speech Enriched with Socio-Demographics for Personality Detection

TLDR
SPADE is introduced, the first dataset with continuous samples of argumentative speech labeled with the Big Five personality traits and enriched with socio-demographic data and conducts feature ablation experiments to investigate which types of features contribute to the prediction of individual personality traits.

References

SHOWING 1-10 OF 108 REFERENCES

Linguistic Features for Readability Assessment

TLDR
Evaluating on two large readability corpora finds that, given sufficient training data, augmenting deep learning models with linguistically motivated features does not improve state-of-the-art performance, and provides preliminary evidence for the hypothesis that the state of theart deeplearning models represent linguistic features of the text related to readability.

ReadNet: A Hierarchical Transformer Framework for Web Article Readability Analysis

TLDR
A new and comprehensive framework which uses a hierarchical self-attention model to analyze document readability, and results show that the proposed method achieves the state-of-the-art performance on estimating the readability for various web articles and literature.

Features Indicating Readability in Swedish Text

TLDR
A study of different levels of analysis and a large number of features and how they affect an ML-system’s accuracy when it comes to readability assessment finds that the best performing features are language models based on part-of-speech and dependency type.

Readability-based Sentence Ranking for Evaluating Text Simplification

TLDR
A new method for evaluating the readability of simplified sentences through pair-wise ranking, which correctly identifies the ranking of simplified and unsimplified sentences in terms of their reading level with an accuracy of over 80%, significantly outperforming previous results.

On Improving the Accuracy of Readability Classification using Insights from Second Language Acquisition

TLDR
It is shown that the developmental measures from Second Language Acquisition research when combined with traditional readability features such as word length and sentence length provide a good indication of text readability across different grades.

Making Readability Indices Readable

TLDR
This work presents a system that, for a given document in Italian, provides not only a list of readability indices inspired by Coh-Metrix, but also a graphical representation of the difficulty of the text compared to the three levels of Italian compulsory education, namely elementary, middle and high-school level.

ReadAid: A Robust and Fully-Automated Readability Assessment Tool

  • Rani QumsiyehYiu-Kai Ng
  • Computer Science
    2011 IEEE 23rd International Conference on Tools with Artificial Intelligence
  • 2011
TLDR
A robust, fully-automated readability analyzer, denoted ReadAid, which employs support vector machines to combine features from the US Curriculum and College Board, traditional readability measures, and the author and subject area of a text document d to assess the readability level of d.

An analysis of a French as a Foreign Language Corpus for Readability Assessment

TLDR
The collection process of an annotated corpus of French as a foreign language texts with the purpose of training a readability model is described and it appears that, for some educational levels, the hypothesis of the annotation homogeneity must be rejected.

A Comparison of Features for Automatic Readability Assessment

TLDR
It is found that features based on in-domain language models have the highest predictive power and Entity-density and POS-features, in particular nouns, are individually very useful but highly correlated.

A Machine Learning Approach to Persian Text Readability Assessment Using a Crowdsourced Dataset

TLDR
The first model for Persian text readability assessment using machine learning was introduced and revealed that this model was accurate and could assess the readability of Persian texts with a high degree of confidence.
...