Transformer-Aided Semantic Communications

The transformer structure employed in large language models, as a specialized category of deep neural networks featuring attention mechanisms, stands out for their ability to identify the most relevant aspects of input data. Such a capability is particularly beneficial in addressing a variety of communication challenges, notably in semantic communication where proper encoding of the relevant data is critical especially in systems with limited bandwidth. In this work, we employ vision transformers specifically for the purpose of compression of the input image, with the goal of preserving semantic information. Through the use of the attention mechanism inherent in transformers, we create an attention mask. This mask effectively prioritizes critical segments of images for transmission, ensuring that the reconstruction phase focuses on key objects highlighted by the mask. Our methodology significantly improves the quality of semantic communication and optimizes bandwidth usage by encoding different parts of the data in accordance with their semantic information content. We evaluate the effectiveness of our proposed framework using the TinyImageNet dataset, focusing on both reconstruction quality and accuracy. Our evaluation results demonstrate that our framework successfully preserves semantic information, by transmitting only a fraction of the encoded data, according to the intended compression rates.