A Feasible Compression Scheme for Bangla Natural Text

The most significant zone of study for Natural Language Processing (NLP) is text compression. Since the number of characters in Unicode are many, 16 bits are needed to process the entire code. This study suggests a useful method for compressing Bangla natural text called the 5 Bit Compression Scheme for Bangla Text(5BCS), which uses a 5-bit encoding method to change the 16-bit Bangla letters into a 5-bit format. This technique provides an encoding algorithm which representing the 16-bit Bangla characters to 5 bits by utilizing a query table. The query table contains 4 sets which comprises all the Bangla characters as well as transliterations and special symbols. An m-series technique is used for logically determining index of a given characters combination while translating characters into 5 bits. By utilizing the 5BCS, a natural text can be compressed by a maximum of 84%. The algorithm for decompress the compressed data to translate the original data is also detailed in this paper. Additionally, a comparison section is developed to compare our 5BCS technique with the widely used Huffman and LZW technique. The experimental result of the comparison looks very much optimistic.