Consideration of parallel data processing over an apache spark cluster

The Spread of cameras and sensors and cloud technologies enable us to obtain life logs at ordinary homes and transmit the captured data to a cloud for life log analysis. However, the amount of processing for video data analysis in a cloud drastically increases when a very large number of homes send data to the cloud. In this research, we aim to improve the efficiency of distributed video data analysis processing by using the parallel deep learning framework Chainer [2] and the distribution processing platform Apache Spark [1] (Spark). In this paper, we construct a Spark cluster and investigate the performance of parallel data processing using Spark varying parameter settings.