Real-Time Vehicle Traffic Analysis using Long Short Term Memory Networks in Apache Spark

Escalating traffic congestion in large and rapidly evolving metropolitan areas all around the world is one of the inescapable problems in our daily lives. In light of this situation, traffic monitoring and analytics is becoming the need of the hour. Real-time traffic analysis requires the processing of data streams that are generated continuously to gain quick insights. In order to processes stream data at a faster rate, we need technologies with high computing capacity. Big data frameworks such as Apache Hadoop, Spark and Kafka, with their capability of processing a massive amount of data, have made it possible to develop advanced and efficient data stream processing systems. The challenge of analyzing data streams for real-time prediction can be overcome by exploiting deep learning techniques. Taking this as a motivation, this work aims at developing a real-time data stream processing model for forecasting vehicle traffic, using Long Short-Term Memory (LSTM) networks to learn and train itself from traffic data. In the proposed analytical framework, the traffic data from an API is streamed using a distributed streaming platform called Kafka into the machine learning model in Apache Spark for analysis. The proposed model is aimed at predicting traffic flow information by integrating Spark and Kafka along with deep neural networks, that will be of great value to the citizens as well as the government by reducing the travel time, cost and energy, thus having a positive impact on the environment and the society.