An Empirical Study on Efficiency of a Dictionary Based Viterbi Algorithm for Word Segmentation

In this paper we present an algorithm for segmenting English sentences, without spaces, into their constituent words based on a dictionary using a variation of the Viterbi algorithm. We call this variation the dictionary based Viterbi algorithm for word boundary identification. The input to the algorithm is a sequence of English words which are not space separated and the output is list of the top-k possible word segmentations. The top-k segmentation of the input into the constituent words is determined on the basis of an optimality criteria applied to a scoring function which computes the "segmentation score" of each possible word sequence for the given input. Our implementation works from the back (right) of the input string and creates a data structure for storing the possible word boundaries during the first pass. The data structure consists of an index (a list of nodes) of length equal to the number of input characters in the string. The i
^th
node in the index contains a pointer to the optimal list of words for the substring starting at position i of the input sequence. The list of possible breakdowns including the optimal one is collected during a second pass through the data structure. Due to the way the initial data structure is built during the first pass we call our algorithm the Reverse Sequence Search (RSS) algorithm. We report experimental results on several datasets including the entire Brown corpus with over one million words to show how fast our algorithm works. We also compare our implementation with a dynamic programming implementation provided by Norvig [1] which is one of the classical algorithms for the problem of word segmentation. Furthermore we show that our algorithm runs in O(n) time and space.