Multimodal Hybrid Approach for Fine-Grained Classification of Diverse Dermatological Conditions

Among the spectrum of dermatological conditions, melanoma stands as the most lethal, emanating from melanocytes-cells responsible for producing melanin, the skin’s pigment. Distinctively more aggressive than its counterparts, melanoma possesses a higher propensity to metastasize, rendering it one of the most formidable types of skin cancer. Given its rapid invasion and propensity for widespread metastasis, prompt and accurate diagnosis, followed by immediate intervention, is imperative. Fine-grained classification tasks are particularly pertinent in this context, aiming to discern between closely resembling subclasses, a challenge akin to differentiating subtle visual nuances among similar-appearing skin pathologies. Consequently, fine-grained classification emerges as a critical tool in the nuanced diagnosis, early detection, and ongoing surveillance of skin diseases. In response, we have developed a innovative network, termed Multimodal Hybrid Approach for Fine-Grained Classification of Diverse Dermatological Conditions (MultiFG), adept at handling a variety of skin conditions. This model proficiently extracts and processes metadata while concurrently capturing visual details through a Convolutional Neural Network (CNN). This is followed by the integration of visual and metadata insights within a Transformer architecture, facilitating enhanced fusion and refined fine-grained analysis. Our proposed methodology’s efficacy is substantiated through rigorous testing on two distinct datasets, where it has demonstrated promising results.