Domain-specific Chinese word segmentation using suffix tree and mutual information

Abstract

As the amount of online Chinese contents grows, there is a critical need for effective Chinese word segmentation approaches to facilitate Web computing applications in a range of domains including terrorism informatics. Most existing Chinese word segmentation approaches are either statistics-based or dictionary-based. The pure statistical method has lower precision, while the pure dictionary-based method cannot deal with new words beyond the dictionary. In this paper, we propose a hybrid method that is able to avoid the limitations of both types of approaches. Through the use of suffix tree and mutual information (MI) with the dictionary, our segmenter, called IASeg, achieves high accuracy in word segmentation when domain training is available. It can also identify new words through MI-based token merging and dictionary updating. In addition, with the proposed Improved Bigram method IASeg can process N-grams. To evaluate the performance of our segmenter, we compare it with two wellknown systems, the Hylanda segmenter and the ICTCLAS segmenter, using a terrorism-centric corpus and a general corpus. The experiment results show that IASeg performs better than the benchmarks in both precision and recall for the domain-specific corpus and achieves comparable performance for the general corpus.

DOI: 10.1007/s10796-010-9278-5

Extracted Key Phrases

6 Figures and Tables

Cite this paper

@article{Zeng2011DomainspecificCW, title={Domain-specific Chinese word segmentation using suffix tree and mutual information}, author={Daniel Dajun Zeng and Donghua Wei and Michael Chau and Fei-Yue Wang}, journal={Information Systems Frontiers}, year={2011}, volume={13}, pages={115-125} }