A Broad-Coverage Normalization System for Social Media Language


Social media language contains huge amount and wide variety of nonstandard tokens, created both intentionally and unintentionally by the users. It is of crucial importance to normalize the noisy nonstandard tokens before applying other NLP techniques. A major challenge facing this task is the system coverage, i.e., for any user-created nonstandard term, the system should be able to restore the correct word within its top n output candidates. In this paper, we propose a cognitivelydriven normalization system that integrates different human perspectives in normalizing the nonstandard tokens, including the enhanced letter transformation, visual priming, and string/phonetic similarity. The system was evaluated on both wordand messagelevel using four SMS and Twitter data sets. Results show that our system achieves over 90% word-coverage across all data sets (a 10% absolute increase compared to state-ofthe-art); the broad word-coverage can also successfully translate into message-level performance gain, yielding 6% absolute increase compared to the best prior approach.

Extracted Key Phrases

8 Figures and Tables

Citations per Year

82 Citations

Semantic Scholar estimates that this publication has 82 citations based on the available data.

See our FAQ for additional information.

Cite this paper

@inproceedings{Liu2012ABN, title={A Broad-Coverage Normalization System for Social Media Language}, author={Fei Liu and Fuliang Weng and Xiao Jiang}, booktitle={ACL}, year={2012} }