NetFlinCS: A hybrid cloud-based framework to allow context-based detection and surveillance

In this paper, we present the NetFlinCS framework, which allows a fast and effective development of compute-intensive interactive applications for mobile devices using hybrid cloud technologies. In addition, we show its advantages by using the proposed technology to build a context-based detection and surveillance application for construction sites, which aggregates all relevant data during architecture planning and the construction process for documentation and later use in facility management. The NetFlinCS framework enables rich Mixed Reality techniques like sensor fusion over multiple mobile devices or AI-based recognition as well as the integration of databases to enable an automatic detection of discrepancies based on common tracking technologies. The device uses the location awareness to process the camera stream in the context of the local planning data. To reduce the client's workload as well as for better scalability and to enable the heavy use of machine learning techniques, we use a hybrid cloud environment that consists of all available servers and clients to outsource expensive calculations. The hybrid cloud backend provides different services like machine learning based segmentation, inpainting, and scene classification that support navigation, video processing, and error estimation in real-time. By deeply integrating standard data formats like the Industry Foundation Classes standard and Building Information Modeling it is possible to compare these databases with tracking data to detect and localize discrepancies over the whole life cycle of a building, from planning data (blueprints, architectural models, etc.) up to the real building.