Road Hazard Detection and Sharing with Multimodal Sensor Analysis on Smartphones

The sensing, computing and communicating capabilities of smart phones bring new possibilities for creating smart applications, including in-car mobile applications for smart cities. However, due to the dynamic nature of vehicles, many requirements such as sensor management, signal and image processing or information sharing needs exist when developing a smart sensor-based in-car mobile application. On the other hand, most in-car applications generally employ single-modal sensor analysis, which also yields limited results. Using the advanced capabilities of smart phones, this study proposes a framework with built-in multimodal sensor analysis capability, and enables easy and rapid development of signal and image processing-based smart mobile applications. Within this framework, an abstraction for fast access to synchronized sensor readings, a plug in based multimodal analysis interface for signal and image processing applications, and a toolset to connect to other users or servers for sharing the results are provided built-in. As part of this study, a sample mobile application is also developed to demonstrate the applicability of the framework. This application is used for detecting defects on the road, such as potholes and speed bumps, and it automatically extracts the video section and the image of the corresponding road segment containing the defect. Upon such critical hazard detection, the application instantly informs nearby users about the incident. A good detection rate of speed bumps is obtained in the performed tests, while the advantage of automatic image extraction based on the multimodal approach is also demonstrated.