Advancing Skin Disease Diagnosis: A Multimodal Approach Utilizing Telegram Api Token Chatbot for Text and Image Analysis in Skin Disease Classification

The human skin serves as a critical indicator of underlying health conditions, often manifesting early signs of disorders affecting internal organs. Recognizing these signs is crucial for timely diagnosis and treatment. However, the significance of the body’s natural defence mechanism through the skin is sometimes overlooked. This study aims to develop a skin disease classification system using a multimodal approach, integrating ensemble methodology using DenseNet169 – Resnet50 ensemble Transfer learning models, and Natural Language Processing (NLP) within a Telegram chatbot interface. The primary aim is to enhance the chatbot’s ability to deliver customised skin-related diagnoses by using user-provided information, including skin type, chemical exposure, and previous treatments, ensuring more precise and personalised interactions. Additionally, the accuracy and generalization capability of the classification system is improved by analysing the data from both the chatbot and image analysis. The chatbot’s self-learning capabilities allow it to improve its comprehension over time in response to user input, which makes it more adept at personalising queries. Integrating DenseNet169 and ResNet50 strengthens the feature reuse by connecting each layer to every other layer in a dense manner for efficient gradient flow thus reducing the number of parameters. ResNet50, with its residual connections, helps mitigate vanishing gradient issues, allowing for deeper networks with stable training leading to improved feature extraction and representation. This hybrid approach captures fine-grained global features enhancing the feature learning attaining good performance in complex tasks. A total of 11,747 images were evaluated in the proposed study, with 7,930 images allocated for training and validation, and 3,817 images designated for testing. The model attained an accuracy of 77.07% and an AUC score of 96.72%. The NLP training model achieved an accuracy of 93.62% ensuring a comprehensive understanding of user data and accelerating engagement, leading to more precise and personalized predictions of skin disorders.