Artificial Intelligence-Driven Cloud-Native Big Data Analytics for Agile Decision-Making in Dynamic Environment

This research focused on real-time decision making in uncertain and volatile business context by creating a cloud-based, big data analytics framework supported by artificial intelligence. To address these issues, this study intended to incorporate stream processing techniques, distributed machine learning algorithms, and cloud service architectures. An architecture based on microservices was adopted with an emphasis on using containers and Kubernetes for supervision. It included Apache Kafka for streaming ingestion, Apache Flink for stream processing, and Apache Spark for batch analysis. Both ensemble methods and deep learning algorithms were used with TensorFlow on Kubernetes. The architecture showed twice the performance gains over traditional approaches in terms of data processing and analysis, while the data ingestion rates were higher by a factor of ten. Machine learning models achieved 94% accuracy in different aspects of prediction, using dynamic learning models to update their models based on current trends in the flow of data. The research was useful for extending the scientific community’s theory in multiple fields, such as integrating deep learning with distributed stream processing, adopting closed-loop control systems for adaptive analytics, and microservices for big data platforms. The study also gave some consideration to ethical issues around decision-making through the AI utilization of explainable AI methods. The research focused on the high level of security addressing techniques, organizing end-to-end encryption, and creating service key management for interaction with hardware security modules. A chaos engineering framework was also adopted into the system to test the overall stability of the system under different failure conditions in order to further strengthen the system. The discoveries indicated that the proposed model effectively improved decision-making in volatile surroundings, which provided ideas for future studies in real-time large dataset applications.