IoT to Structured Data (IoT2SD): A Big Data Information Extraction Framework

Devices characterized as Internet of Things (IoT) have proven to be very useful in a myriad of business sectors. The current success in being able to have multiple small-footprint devices working independently or in some aggregate manner is drawing more attention as a solution to many industries. Security, speed, and unstructured data are some of the challenges within the scope of Big Data and IoT. This data stream, ultimately making its way to a data store, is typically semi-structured or unstructured data. In this state, it is not easily searchable or it is not easy to have computational operations performed on it. This prevents real-time or near real-time data analysis tools from working on the data store, transforming the newly arrived data into actionable information. In this research, the IoT to Structured Data (IoT2SD) Information Extraction framework is proposed; it incorporates an Intrusion Detection System (IDS) to protect a NoSQL data store, to allow the processing of non-attack (benign) network flows to structure unstructured IoT MQTT message data. The framework positions targeted MQTT data for the end purpose of data analytics. In addition, this work demonstrates the feasibility of the IoT2SD Information Extraction framework to function as an online solution.