Image Generation Reflecting the Meaning of Language That Reveals Object’s Attributes

Although recent Text-to-Image models have achieved great success on generating images from the description of an object such as “a bird with brown and black striped wings and a yellow beak” these models still struggle to generate images based on the understanding of the attributes of the object. In this study, we aim to learn the correspondence between the direction in which the characteristics of an object are made more apparent by language, e.g., adjectives, and the direction of shape change of the object, and to generate images with shape changes that emphasize the characteristics of objects by words. As a concrete experiment, using images of shoes as the subject, we constructed an embedded space of variational autoencoders for the three shoe categories Shoes, Boots, and Sandals that reflect the meanings of the four adjectives: “open”, ‘pointy”, “sporty”, and “comfort” as their properties. By moving the embedding vectors of the shoes in the direction that reveals the meaning of the four adjectives, we generated images that reflect the meaning of words in the characteristics of the shoes, e.g., “more sporty Boots”.