Twitter Part-of-Speech Tagging for All: Overcoming Sparse and Noisy Data


Part-of-speech information is a pre-requisite in many NLP algorithms. However, Twitter text is difficult to part-of-speech tag: it is noisy, with linguistic errors and idiosyncratic style. We present a detailed error analysis of existing taggers, motivating a series of tagger augmentations which are demonstrated to improve performance. We identify and evaluate techniques for improving English part-of-speech tagging performance in this genre. Further, we present a novel approach to system combination for the case where available taggers use different tagsets, based on voteconstrained bootstrapping with unlabeled data. Coupled with assigning prior probabilities to some tokens and handling of unknown words and slang, we reach 88.7% tagging accuracy (90.5% on development data). This is a new high in PTB-compatible tweet part-of-speech tagging, reducing token error by 26.8% and sentence error by 12.2%. The model, training data and tools are made available.

Extracted Key Phrases

11 Figures and Tables

Citations per Year

118 Citations

Semantic Scholar estimates that this publication has 118 citations based on the available data.

See our FAQ for additional information.

Cite this paper

@inproceedings{Derczynski2013TwitterPT, title={Twitter Part-of-Speech Tagging for All: Overcoming Sparse and Noisy Data}, author={Leon Derczynski and Alan Ritter and Sam Clark and Kalina Bontcheva}, booktitle={RANLP}, year={2013} }