AI-Enabled Learning for Imbalanced Class Data using Spark Streaming Concepts

Chronic Diseases are a very significant problem around the world. Manhandling medical cases often leads to significant repercussions in the treatment process and needs to be detected and dealt with early on. Unusual clinic actions are often related to medical errors, enabling detection and alerting. Huge leaps in machine learning, data science, and big data analytics techniques have led to innovative ways to deal with these issues. As a result, mortality rates can be greatly reduced in all sorts of medical cases. In this work, a new ensemble method is proposed for multiclass classification applied on the US-Chronic Disease Indicator (US-CDI) dataset and validated using UCI-Diabetes data by applying K-means clustering to it to detect chronic disease. The proposed hybrid model, KD-Boost Ensemble, is based on the concept of ensemble voting mechanisms, which combine the predictions of the K-neighbors classifier, Decision Tree classifier, and Gradient Boosting classifier to obtain improved accuracy over the dataset when compared to the individual conventional classifiers. This chronic disease classification achieved 96% accuracy using the proposed model. The devised model is validated through the diabetes dataset in a real-time environment and attains 98% accuracy.