A Patch-Based Saliency Detection Method for Assessing the Visual Privacy Levels of Objects in Photos

Photo privacy protection has recently received increasing attention from the public. However, the overprotection of photo privacy by hiding too much visual information can make photos meaningless. To avoid this, visual information with different degrees of privacy sensitivity can be filtered out using various image-processing techniques. Objects in a photo usually contain visual information that can potentially reveal private information; this potential depends on both the visual saliency of the objects and on the specific categories to which the objects belong. In this paper, we aim to quantitatively evaluate the influence of visual saliency information on privacy and objectively evaluate the levels of visual privacy that objects contain. Meeting this objective faces two challenges: 1) determining a method of effectively detecting generic objects in a photo for the extraction of saliency information and 2) determining a scientific method for assessing the visual private information contained in objects. To cope with these challenges, we first propose a hierarchical saliency detection method that combines a patch-based saliency detection strategy with an objectness estimation strategy to effectively locate salient objects and obtain the saliency information of each object. The proposed method results in a small set of class-independent locations with high quality and a mean average best overlap score of 0.627 at 1150 locations, which is superior to the score of other saliency detection methods. Second, we build a computational privacy assessment system to scientifically calculate and rank the privacy risks of objects in a photo by creating an improved risk matrix and using the Borda count method. The proposed computational privacy assessment method matches human evaluations to a relatively high degree.