Type-Token Analysis on LDC-IL Text Corpus

Large quantity of text data is essential to develop natural language processing applications. This paper describes the statistical analysis of LDC-IL text data. Type token analysis is carried out on 8 languages of a substantial size to check for closure. The study shows how similar language or same language family languages are giving an equivalent pattern of saturation curve.