Evaluation of Change Detection Algorithms using Difficulty Maps

The evaluation of a change detection algorithm should show its superiority over state-of-the-art algorithms' performances. Evaluating an algorithm involves executing it to segment a set of videos and comparing the results with the ground truth. Here, we used the difficulty level to classify each pixel of each frame of the videos of a dataset as an algorithm performance measure. A structure called "difficulty map" stores information about the difficulty of classifying each pixel in a frame. Based on these maps, we developed a metric that aims to evaluate the performance of algorithms on the difficulty map. The results showed that there are algorithms with the characteristic of classifying pixels that most state-of-the-art algorithms cannot classify (promising algorithms). Identifying such algorithms is essential since improving their performance means facing challenges already overcome by existing approaches.