Twitter Part-of-Speech Tagging for All: Overcoming Sparse and Noisy Data

Abstract

Part-of-speech information is a prerequisite in many NLP algorithms. However, Twitter text is difficult to part-of-speech tag: it is noisy, with linguistic errors and idiosyncratic style. We present a detailed error analysis of existing taggers, motivating a series of tagger augmentations which are demonstrated to improve performance. We identify and evaluate techniques for improving English part-of-speech tagging performance in this genre. Further, we present a novel approach to system combination for the case where available taggers use different tagsets, based on vote-constrained bootstrapping with unlabeled data. Coupled with assigning prior probabilities to some tokens and handling of unknown words and slang, we reach 88.7% tagging accuracy (90.5% on development data). This is a new high in PTB-compatible tweet part-of-speech tagging, reducing token error by 26.8% and sentence error by 12.2%. The model, training data and tools are made available.

Extracted Key Phrases

Showing 1-10 of 34 references

Note on the sampling error of the difference between correlated proportions or percentages

Highly Influential
1 Excerpt
0204020132014201520162017
Citations per Year

103 Citations

Semantic Scholar estimates that this publication has received between 73 and 151 citations based on the available data.

See our FAQ for additional information.