Weather data analysis using spark — An in-memory computing framework

Weather forecasting is a motivational challenge in human civilization. There is a shift in scientific focus towards solving this problem. Weather is a complex phenomenon that involves dynamic interaction of several forces. There are number of numerical weather models and algorithms that have been developed and enforced to predict the weather forecasting. Weather forecasting plays a very crucial role in various fields like agriculture, government planning, industries, predicting stock analysis etc. Various sensors are deployed at different geographical locations to collect weather data on a daily basis. The greatest challenge is to store and analyze the huge volume of data in an effective manner. Big data technologies are used to address these issues and challenges using distributed computing. Hadoop MapReduce is used to analyze the weather data. But spark is an emerging technology that performs in-memory computing which is more efficient than Hadoop mapreduce. This paper proposes the spark implementation for weather data analysis and shows the highest average precipitation, temperature or any weather related parameters for different weather stations.