PD-UVU: Prompts-Driven Unified Visual Understanding Model Based on Specific Queries

All visual understanding tasks aim to locate specified objects based on language expressions, category names, or annotation information as queries. Currently, tasks with this common feature are completed independently as multiple sub-tasks. In this paper, we propose a universal visual understanding model, PD-UVU. PD-UVU redefines diverse visual understanding tasks as a unified prompts-driven query paradigm, where changing the prompts can retrieve objects of a specific category. This unified representation brings the following benefits: (1) a universal object-centered semantic representation can be trained based on a large amount of data from different visual tasks and label vocabularies, which is particularly advantageous for tasks with limited training data; (2) a unified visual understanding model has strong parameter efficiency and domain generalization ability, enabling the simultaneous handling of multiple tasks and saving redundant computations required by small models for independent tasks. PD-UVU shows outstanding performance on 20 benchmarks from 10 challenging visual understanding tasks, including image-level tasks such as object detection and instance segmentation, expression understanding and language-image segmentation tasks, as well as object tracking tasks.