Robust near-duplicate video retrieval using compressed representation

Recently the amount of videos produced by mobile devices has grown, as well as the variety of video analytics services, able to perform on-device classification, automated tagging, video retrieval, object tracking and similarity analysis. However, high computational complexity limits their use in real-time applications (such as, objects detection in Closed Circuit Television (CCTV) system), may introduce poor user experience because of delays or may have privacy nuances because of video content sharing with edge computing services. Because of that, of special interest are approaches that use advanced hardware Artificial Intelligence (AI) features of modern mobile platforms, such as Qualcomm AI Engine, Samsung’s AI Quantum Processor, MediaTek’s AI Processing Units etc. Online services for video search by an (optionally distorted) video fragment do exist, but they are very limited and bound to cloud-side processing. This inspired us to develop a core technology for near-duplicate video retrieval that would allow such a video search service to operate efficiently, be applicable for on-device execution and enable many other practical applications (e.g. video provenance, authenticity check or recommendation service). We propose an adaptation of modern Visual Transformer (VT) model for processing partially decompressed video stream and matching a small sets of reference frames (keyframes) with a pristine reference video. The adaptation of novel DINO model [3] in our proposed approach allows to improve the detection of near-duplicates by 20% (from 66.1% to 88.3%) in comparison with the state-of-the-art DnS [13] and S2VS [12] models. The proposed models are robust against popular video content modifications, such as affine transformations and visual effects, even with video transcoding by novel Essential Video Coding (EVC) codec. Also, our proposed solution makes it possible to shorten processing duration down to 2.5 times in comparison with the approach the requires full video decompression. Low computational complexity allows our technology for on-device use.