Using Semantic Segmentation to Assist the Creation of Interactive VR Applications

The creation of interactive VR applications from 3D scanned content usually includes a lot of manual and repetitive work. Our research aim is to develop a real-world, cross-domain, automatic, semantic segmentation system that enhances the creation of interactive VR applications. We trained segmentation agents in a superpoint growing environment that we extended with an expert function. This expert function solves the sparse reward signal problem of the previous approaches and enables to use a variant of imitation learning and deep reinforcement learning with dense feedback. Additionally, the function allows to calculate a performance metric for the degree of imitation for different segmentations. We trained our agents with 1182 scenes of the ScanNet data set. More specifically, we trained different neural network architectures with 1170 scenes and tested their performance with 12 scenes. Our intermediate results are promising such that our segmentation system might be able to assist the VR application development from 3D scanned content in near future.