Representing Character Sequences as Sets : A simple and intuitive string encoding algorithm for NLP data cleaning

Data preparation is vital in the field of machine learning, especially in natural language processing. This paper briefly describes a method of encoding character sequences, which is inspired by biological memory systems. The algorithm can be thought of as a type of artificial neural network, which is only used to store text data in a specific way, not generate predictions. To test the viability of the approach, the task of searching for words in a dictionary was used, utilizing two simple heuristic rules. The goal is to produce tools capable of cleaning texts written in any language, which uses a phonemic writing system. Texts written in two specific languages are being processed during development and experimentation, they are English and Bulgarian. User-friendliness and minimization of data obfuscation are the driving requirements. Another requirement is compatibility with other machine learning methods, since the algorithm is not intended for solving an entire NLP case from start to finish. It is to be used only during the data preparation phase.