Point-Whisper: Boosting Pixel-to-Point via Dimensional Knowledge Uncovery and Efficient Enhancement Tuning

Within the sphere of deep learning, current discourse underscores the relevance of pre-trained expansive models on large-scale datasets. These pre-trained models, distinguished by their extraordinary representational prowess and transferability, have demonstrated notable success in multiple downstream tasks related to natural language processing and 2D vision. However, the adaptation of this pre-trained model to 3D vision presents significant hurdles due to the scarcity and difficulty associated with data collection. To alleviate the issues, we introduce the method, termed as Point-Whisper, leveraging extant 2D knowledge and applying it to the 3D domain. Specifically, we advocate for the use of the Decoupled and Enhancement technique to refine pre-trained image models for point cloud comprehension. We unlock the pre-trained image models' hidden 3D point-friendly knowledge via dimensional information, optimize these masks on the downstream data with the parameters frozen. Additionally, we integrate the LoRA tuning to augment the point cloud-specific knowledge, thereby enhancing the representation of 3D vision. The experiment results indicate that our method substantially boosts performance by a significant margin.