Comparative Analysis of Bengali Stop Word Detection Using Different Approaches

Stop words are employed for sentence formation without contributing semantic content in any language. In the field of text mining, information retrieval, text summarizer, indexing, search, and retrieval systems, and so on, stop words must be excluded. Until now, very few works have been done for Bengali stop word detection and handle the unique problem of stop words acting as content words contextwise and vice-versa. So, in this paper different approaches are the Dictionary-based approach, POS Tagging based approach, Word2Vec based approach, and FastText based approach, is used and analyzed with 20,000 Bengali sentences. The dictionary-based approach outperformed with 92 percent accuracy, indicating that adopting the dictionary-based approach for Bengali stop word detection and solving the specific case problem can result in an automatic, reliable, and functioning system. Also applying Word2Vec and FastText based approach is the first attempt for Bengali stop word detection in Bengali language processing.