SketchTriplet: Self-Supervised Scenarized Sketch–Text–Image Triplet Generation

Touchscreen Internet of Things (IoT) devices, such as smartphones and tablets, have been seamlessly integrated into our daily lives. Drawing sketches on the touch screen is an extremely convenient mode of interaction, and when combined with generative artificial intelligence (AI), it makes the customized data generation and digital twin based on IoT devices more straightforward. However, the scarcity of free-hand sketch data makes the construction of data-driven generative AI models a thorny issue. Despite the emergence of some large-scale sketch datasets, these datasets primarily consist of sketches at the single-object level. There continues to be a lack of large-scale paired datasets for scene sketches. In this article, we propose a self-supervised method for scene sketch generation that does not rely on any existing scene sketch, enabling the transformation of single-object sketches into scene sketches. To accomplish this, we introduce a method for vector sketch captioning and sketch semantic expansion. Additionally, we design a sketch generation network that incorporates a fusion of multimodal perceptual constraints, suitable for application in zero-shot image-to-sketch downstream task, demonstrating state-of-the-art performance through experimental validation. Finally, leveraging our proposed sketch-to-sketch generation method, we contribute a large-scale dataset centered around scene sketches, comprising highly semantically consistent “text-sketch–image” triplets. Our research can provide critical technical support for AI content creation and customized data generation based on IoT devices.