Transformer-based single view joint prototype features adversarial reconstruction of probabilistic point cloud

Inferring the three-dimensional shape of an object from a single image is a highly challenging and uncertain problem. This paper proposes a transformer-based single-view adversarial reconstruction point cloud neural network (TP-GAN) aimed at fusing 3D point cloud prototype features with 2D image features to generate a point cloud from a single image. Firstly, the network learns a set of 3D prototype features from an objective point cloud dataset and dynamically adjusts them through training. Secondly, these prototypes are integrated with incoming image features to guide the point cloud generation process. Finally, a generative adversarial structure is introduced to improve the generated point cloud. Additionally, to address the uncertainty of the self-obscuring part of the object, TP-GAN can learn Gaussian probability distributions from the feature vectors to predict multiple plausible point clouds. The proposed TP-GAN is evaluated on ShapeNet, ModelNet40, and Pix3D datasets, and extensive experiments demonstrate that it outperforms the state-of-the-art method (SOTA) in the singleview 3D reconstruction task.