Adaptive Buffering Strategies for Incremental Learning Under Concept Drift in Lifestyle Disease Modeling

Lifestyle diseases such as diabetes manifest through subtle and non-stationary clinical patterns, posing significant challenges for real-time prediction and monitoring. Conventional machine learning models often struggle to maintain performance under evolving data distributions due to concept drift. This study proposes an adaptive deep learning framework designed to handle concept drift through incremental learning in clinical data streams. The investigation centers on evaluating the effectiveness of various buffering strategies namely, adaptive buffering, FIFO buffering, and streaming without buffering combined with drift detection mechanisms for healthcare prediction. A balanced clinical dataset exhibiting evolving patterns was used to benchmark model performance. Deep learning architectures, including BiLSTM, GRU, LSTM, and Bayesian Neural Networks were incrementally trained, and their drift(Abrupt, Gradual and Recurring) responsiveness was assessed using three categories of detectors: 1) statistical test-based, 2) error-rate-based, and 3) uncertainty-based approaches leveraging Monte Carlo Dropout. Results indicated that adaptive buffering strategies consistently outperformed FIFO and no-buffer strategies, yielding higher accuracy, precision, and recall, especially under abrupt and large drift magnitudes. The hybrid drift detection method, combined with Bi-LSTM, demonstrated the best performance in maintaining retention and minimizing forgetting, even as the drift magnitude increased. Additionally, the drift magnitude study highlighted that larger drifts had a significant impact on model performance, with adaptive buffering and uncertainty-based drift detection proving to be more resilient to high drift intensities. This research underscores the importance of combining robust drift detection methods and adaptive buffering strategies to enhance the robustness of models dealing with concept drift in real-world applications.