Image Tagging by Fine-tuning Class Semantics Using Text Data from Web Scraping

The image tagging task aims to assign relevant known tags to an image. It is an active research topic in computer vision and machine learning because of the diversity of its applications in semantic search and image retrieval. Earlier efforts on image tagging address this problem as a multi-level classification problem using visual features from images and semantic word vectors of tags. In most cases, a pre-trained language model like word2vec or Globe is used to obtain those word vectors. Because of using a pre-trained language model, an image tagging approach cannot scale itself to the context of the targeted application. This paper fine-tunes a language (BERT) model using text descriptions obtained from web (Wikipedia) scraping to learn a rich distributed representation of tags. Then, we employ word vectors of tags extracted from finetuned language (BERT) model to solve the image tagging task. Our method is more specialized to the particular application by incorporating context information between targeted tags and images. As a result, word vectors obtained from the fine-tuned model perform better than those from pre-trained language models. We evaluate our method on the widely used NUS-WIDE dataset and achieve competitive results compared with state-of-the-art methods.