Semantic Type Detection or Unlabelled Dataset

Efficient categorization, analysis, and extraction of insights from large textual datasets require the identification of semantic types. Natural language processing (NLP) techniques offer valuable tools, including regex matching, cosine similarity, and range-based classification, to determine the type of information contained within the text. The HPCC’s powerful computing capabilities enable faster execution of complex NLP algorithms, allowing for comprehensive analysis and categorization of extensive datasets. This research aims to propose a methodology that utilises NLP techniques for predicting the semantic type of columns in extensive datasets. Additionally, the accuracy of the predicted types is validated through experimentation. The proposed approach leverages various NLP techniques to predict the semantic type of columns. Regex matching is used for identifying specific textual patterns associated with distinct semantic types. Cosine similarity is employed to measure the similarity between columns and known semantic types, enabling the assignment of appropriate labels based on the most closely related types. Range-based classification is also wherein predefined ranges of values associated with different semantic types are used to allocate labels to columns based on their values. By combining these techniques we obtain a comprehensive and robust framework for semantic type prediction in textual datasets. To evaluate the effectiveness of the proposed methodology, it is tested on multiple datasets. The results obtained demonstrate the reliability of the methodology in predicting semantic types. The research achieves significant advancements in simplification of data categorization, enabling easier extraction of insights from large textual datasets. The methodology’s predictive accuracy ensures that the assigned semantic types are reliable and suitable for subsequent data processing tasks.