Image retrieval for visual understanding in dynamic and sensor rich environments

Vision is vital to decision making, as humans naturally trust their eyes to enhance situation awareness. Yet the modern age has overwhelmed humans with massive amounts of visual information, which is problematic in time sensitive and mission critical situations, such as emergency management and disaster response. More efficient search and retrieval systems address some of these issues, which is why many seek to develop and extend Content Based Image Retrieval (CBIR) techniques to support situational awareness in a more autonomous fashion. However, there is currently no adequate system for CBIR to support situational awareness in dynamic and sensor rich environments. This research proposes an extensible framework for CBIR to support a holistic understanding of the environment through the automated search and retrieval of relevant images and the context of their capture. This constitutes assisted CBIR as embodied in the multi-sensor assisted CBIR system (MSACS). We design the MSACS framework and implement the core CBIR system of MSACS using the state of the art Bag of Visual Words paradigm. The system is evaluated using a dataset of GPS tagged images to show favorable precision and recall of spatially related images. Applications for localization and search for Wi-Fi access points demonstrate improved situational awareness using the system. Assisted CBIR could enable vision based understanding of an environment to ease the burdens of information overload and increase human confidence in autonomous systems.