Wormingo: a 'true gamification' approach to anaphoric annotation

  title={Wormingo: a 'true gamification' approach to anaphoric annotation},
  author={Doruk Kicikoglu and R. Bartle and Jon Chamberlain and Massimo Poesio},
  journal={Proceedings of the 14th International Conference on the Foundations of Digital Games},
In this paper we present Wormingo, 1 a new Game-with-a-Purpose for anaphoric annotation. It introduces the motivation-annotation paradigm which uses linguistic puzzles and other widely known gamification techniques and word game mechanics to motivate players to carry out anaphoric annotation tasks. In a preliminary experiment, the game was tested on 270 players recruited through the Reddit platform, achieving promising results. 
Rigor Mortis: Annotating MWEs with a Gamified Platform
Rigor Mortis is a gamified crowdsourcing platform designed to evaluate the intuition of the speakers, then train them to annotate multi-word expressions in French corpora using some of the tests developed in the PARSEME-FR project. Expand
Aggregation Driven Progression System for GWAPs
This paper presents the progression technique implemented in Wormingo, an NLP GWAP that currently includes two layers of task complexity, and compared the accuracy and engagement achieved with each scenario. Expand
Demonstration of a Serious Game for Spoken Language Experiments - GDX
This work presents a novel experimental framework for forced choice categorization tasks or speech perception studies in the form of a computer game, based on the Unity Engine – the Gamified Discrimination Experiments engine (GDX). Expand


Phrase Detectives: A Web-based collaborative annotation game
The first version of Phrase Detectives is presented, to the authors' knowledge the first game designed for collaborative linguistic annotation on the Web and applying this method to linguistic annotation tasks like anaphoric annotation. Expand
Crowdsourcing Complex Language Resources: Playing to Annotate Dependency Syntax
It is shown that with suitable mechanisms (decomposition of the task, training of the players and regular control of the annotation quality during the game), it is possible to obtain annotations whose quality is significantly higher than that obtainable with a parser. Expand
Making people play for Lexical Acquisition with the JeuxDeMots prototype
This paper proposes a novel approach by making people play some king of associative word games, which can memorize associations where a pair of players agrees and presents the principles of the game and some insights of the data collected so far. Expand
Phrase detectives: Utilizing collective intelligence for internet-scale language resource creation
An overview of all aspects of Phrase Detectives, from the design of the game and the HLT methods the authors used to the results they have obtained so far, and summarizes the lessons that have been learned in developing this game which should help other researchers to design and implement similar games. Expand
Play the Language: Play Coreference
This paper provides a description of the game design that covers the strategy, the instructions for the players, the input texts selection and preparation, and the score evaluation of the PlayCoref game. Expand
Linguistic and Cognitive Evidence About Anaphora
  • Massimo Poesio
  • Psychology, Computer Science
  • Anaphora Resolution - Algorithms, Resources, and Applications
  • 2016
This chapter discusses evidence about how anaphoric expressions are recovered in context, and includes a description of the main models of local focus, distinguishing between discrete and activation-based approaches. Expand
It’s All Fun and Games until Someone Annotates: Video Games with a Purpose for Linguistic Annotation
This work proposes a new linguistic annotation paradigm that produces annotations from playing graphical video games that yields annotation quality equal to that of experts and a cost reduction of 73% over equivalent crowdsourcing. Expand
Gamification for Word Sense Labeling
This work shows how gold standard data for word sense disambiguation can be obtained using a “Game with a Purpose” (GWAP) called Wordrobe, which consists of a large set of multiple-choice questions on word senses generated from the Groningen Meaning Bank. Expand
"Dr. Detective": combining gamification techniques and crowdsourcing to create a gold standard in medical text
A design for a gamified crowdsourcing workflow to extract annotation from medical text to resolve the main problem in gathering ground truth from experts - that the low inter-annotator agreement is typically caused by different interpretations of the text. Expand
The Magic Circle and the Puzzle Piece
In a common description, to play a game is to step inside a concrete or metaphorical magic circle where special rules apply. In video game studies, this description has received an inordinate amountExpand