Self-Attention Based Image Feature Representation for Medical Visual Question Answering

Medical Visual Question Answering is an emerging field which uses artificial intelligence to effectively combine the multimodal data and generate output. Users interact with the visual data through text to generate answers, which may not always be directly accessible from the image. In this paper, a network has been proposed that aims to generate a comprehensive image feature representation, capturing the most relevant visual information present in the image, which is essential for effectively answering questions. The image features are extracted using Uniform Local Binary Pattern Histogram and Gray Level Co-occurrence Matrix, which are then combined using the multihead self-attention mechanism obtaining a compact and meaningful representation. This combined representation is further fused with deep features to get the desired image feature embedding. Multimodal fusion of the image and text feature embeddings generated the final representation required for the downstream task of Visual Question Answering.