Masked Generative Extractor for Synergistic Representation and 3D Generation of Point Clouds

Representation and generative learning mutually enhance each other across various domains. However, in point cloud processing, these models are typically trained independently, failing to leverage their synergy. To integrate 3D representation and generative learning deeply, we propose the Point-Mge framework. This framework uses a vector quantized variational autoencoder to reconstruct an occupancy neural field of 3D shapes, learning discrete semantic features of point patches. We introduce sliding masking ratios to smoothly transition from representation to generative learning. Additionally, our method exhibits good generalization in high-capacity models and achieves new state-of-the-art results in multiple down-stream tasks. In shape classification, Point-Mge attained 94.3% (+1.1%) on the ModelNet40 dataset and 92.9% (+5.5%) on the ScanObjectNN dataset. Experimental results also demonstrate that Point-Mge can generate high-quality 3D shapes in both unconditional and conditional settings. Code is available at https://github.com/robhlzeng/Point-Mge.