Distributed Concept Drift Detection for Efficient Model Adaptation with Big Data Streams

Predictive models are essential in big data platforms to tackle the needs of several real-world applications. However, static models are known to be prone to obsolescence in dynamic environments. While concept drift detection represents a viable way to deal with this problem, it is scarcely explored in the context of big data streams. In this paper, we propose a distributed drift detection workflow based on the DDM algorithm paired with a predictive model. Our workflow updates the predictive model as soon as drifts are detected, adjusting to the most recent data characteristics. To enable the analysis of large-scale datasets, we leverage Pandas UDFs and Apache Spark, efficiently distributing this workload across multiple worker node instances. Our experiments on two real-world drift detection datasets show the positive results obtained in terms of Speedup, Scaleup, and a limited impact in detection delay in comparison to a single worker node instance.