An Automatically Learning PSOWTP-CWS Algorithm Based on Word Transition Probability

Most of the available papers on the automatically learning Chinese word segmentation algorithms based on particle swarm optimization (PSO) did not employed PSO as the core algorithm for CWS but as the parameter optimizer. We proposed a novel PSO-cored CWS schema considering the word transition probability (PSOWTP-CWS) and tested the practicability on the bakeoff 2005 dataset. Firstly, the content is split into a mass of non-punctuation clauses. Then PSOWTP-CWS generates the PSO swarm in the segment searching space. Led by the self-knowledge and the social knowledge on the word transition probability, PSOWTP-CWS covers the best segmentation schema with the highest fitness. Experiment results presents that PSOWTP-CWS achieved the acceptable accuracy of Chinese word segmentation and the practical efficiency.