Design Graph Guided Element Importance-Aware Layout Generation With Multimodality Cascade Transformer

Graphic designs are pervasive in our daily lives and widely used to communicate information hierarchically to humans. To achieve this, the layout plays an essential role in guiding readers to understand the importance of different elements and comprehend the content. To deal with the rapidly increasing demands of graphic designs, recent studies attempt to automatically generate layouts based on category information and spatial relations, often resulting in layouts with poor communication quality. In this article, we make the first attempt to explore element importance-aware layout generation under the guidance of a novel design graph, which attracts readers’ attention to a layout by formulating aesthetic relations implicitly involved in graphic designs between element pairs. The core of our approach is a learning-based framework with a new multimodality cascade transformer (MCT) in a coarse-to-fine manner. A hierarchical multimodality fusion (HMF) mechanism and two new losses are introduced to guide the training process progressively. We further collect a new fine-grained advertisement poster layout dataset containing more than 30 K layouts labeled with 91 element labels. Both qualitative and quantitative experiments demonstrate the effectiveness of our approach against existing works. We also conduct user studies and cognitive experiments to evaluate the direct adaptability and attractiveness of generated layouts.