RefAR: 3D Sketch-Based Modeling with In-situ References

3D sketch-based modeling in Virtual Reality (VR) or Augmented Reality (AR) enables users to create 3D content more freely and intuitively. However, mid-air sketching always has poor accuracy and is hard to control the scale, which hinders the free creation of desired models. In this paper, we propose RefAR, a novel in-situ interactive 3D modeling system in AR, that allows users to intuitively draw 3D strokes in mid-air by referring to physical objects in the users' environment and in-situ captured point clouds in real-time using a Microsoft HoloLens 2. The system is designed for novice users to construct desired 3D models from several rough 3D sketches with the help of in-situ objects that act like canvases and moulds. The users can easily and intuitively sketch 3D strokes around the physical objects in AR, and the rough strokes are optimized by fitting them to the captured point clouds, which in turn generate a refined 3D model. We demonstrated the ease of use and effectiveness of RefAR for novice users by creating various man-made object models.