Learning Assisted Interactive Modelling with Rough Freehand 3D Sketch Strokes

Freehand 3D sketches are a great medium to ideate and create visual content. However, generating 3D models from such rough sketches remains an unsolved, non-trivial task. We present an end-to-end interactive framework for rapid, incremental modelling from sparse, irregular 3D sketches. At the core of our solution, is sketchTransformer, a twostaged transformer network architecture, that fits parametric surface patches to a set of sketch strokes. We devise a novel pseudo height field representation that enables the sketchTransformer to handle noise and sparseness in the input strokes. Our method interactively evolves the surface model while maintaining smooth joins between nearby patches. We implement two frontends for our framework, one on the desktop and another as a mobile AR application, to illustrate how our method complements a standard 3D modelling pipelines. Our framework robustly handles a large variety of input 3D strokes that competing methods cannot parse adequately.