HOC-Search: Efficient CAD Model and Pose Retrieval From RGB-D Scans

We present an automated and efficient approach for retrieving high-quality CAD models of objects and their poses in a scene captured by a moving RGB-D camera. We first investigate various objective functions to measure similarity between a candidate CAD object model and the available data, and the best objective function appears to be a ”render-and-compare” method comparing depth and mask rendering. We thus introduce a fast-search method that approximates an exhaustive search based on this objective function for simultaneously retrieving the object category, a CAD model, and the pose of an object given an approximate 3D bounding box. This method involves a search tree that organizes the CAD models and object properties including object category and pose for fast retrieval and an algorithm inspired by Monte Carlo Tree Search, that efficiently searches this tree. We show that this method retrieves CAD models that fit the real objects very well, with a speed-up factor of 10$\times$ to 120$\times$ compared to exhaustive search. We used our method to automatically retrieve CAD models for objects in the ScanNet dataset; our annotations are available at https://github.com/stefanainetter/SCANnotateDataset.