Optimizing Big Data Analytics Architecture for Edge Computing Using Container Technology

This study is dedicated to the integration of big data analytics with edge computing, a critical need driven by the exponential growth of Internet of Things (IoT) technologies and smart device data. We introduce an optimized architecture that integrates Docker containerization with Apache Hadoop YARN, designed specifically for decentralized, resource-constrained edge environments. This innovative framework not only facilitates dynamic resource management but also ensures robust fault tolerance under the heterogeneous and often unstable network conditions typical of edge settings. By integrating Docker and Docker Swarm for container orchestration along with YARN for resource scheduling, our architecture significantly enhances data processing localization. This improvement reduces computational latency and boosts system performance. Through empirical evaluation, we demonstrate marked enhancements in task flexibility, resource efficiency, and operational continuity, particularly adept at managing the diversity and sporadic conditions of edge nodes. Additionally, the deployment of advanced load-balancing tools and multiple management nodes within containers enables the system to autonomously reallocate tasks during node failures or resource-constrained periods. Experimental results show that our architecture achieves an average failover recovery time of less than five seconds and maintains an average CPU utilization below 3% for management nodes, demonstrating significant advantages in handling large-scale IoT data at edge nodes.