DDoS Mitigation while Preserving QoS: A Deep Reinforcement Learning-Based Approach

The deployment of 5G networks has significantly improved connectivity, providing remarkable speed and capacity. These networks rely on Software-Defined Networking (SDN) to enhance control and flexibility. However, this advancement poses critical challenges including expanded attack surface due to network virtualization and the risk of unauthorized access to critical infrastructure. Since traditional cybersecurity methods are inadequate in addressing the dynamic nature of modern cyber attacks, employing artificial intelligence (AI), and deep reinforcement learning (DRL) in particular, was investigated to enhance 5G networks security. This interest arises from the ability of these techniques to dynamically respond and adapt their defense strategies according to encountered situations and real-time threats. Our proposed mitigation system uses a DRL framework, enabling an intelligent agent to dynamically adjust its defense strategies against a range of DDoS attacks, exploiting ICMP, TCP SYN, and UDP, within an SDN environment designed to mirror real-life user behaviors. This approach aims to maintain the network’s performance while concurrently mitigating the impact of the real-time attacks, by providing adaptive and automated countermeasures according to the network’s situation.

I. Introduction

The advent of 5G networks has ushered in a new era of connectivity, promising unprecedented speed, reliability, and capacity [1], [2]. These networks rely on Software-Defined Networking (SDN) as the backbone architecture, serving as the infrastructure that enhances both control and flexibility [1]. However, with this technological leap comes the inevitable challenge of securing these networks against evolving cyber threats. Traditional cybersecurity approaches, such as signature-, blockchain- and rule-based systems, are unable to address the evolved techniques of modern cyber threats as they rely on static and rigid rules, which can result in their inability to effectively deal with unseen attack patterns. Additionally, they may introduce latencies that are unacceptable for real-time applications. In response to these challenges, there is a growing interest in leveraging Machine Learning (ML) and Reinforcement Learning (RL) techniques to enhance the security of 5G networks [3] as they can autonomously discover effective defense strategies in dynamic and uncertain environments. Additionally, RL can iteratively improve its performance over time by adjusting its strategies based on the environment’s feedback, enhancing the resilience of the 5G networks against emerging threats. To provide a comprehensive understanding, we survey relevant literature on the role of ML and Deep Reinforcement Learning (DRL) in enhancing the security of 5G networks including various perspectives and methodologies. In particular, we focus on defenses against distributed denial-of-service (DDoS) attacks given their severe threat to 5G infrastructures. These attacks can control a large number of compromised devices to amplify their malicious impact, making it harder for defenders to simply drop or disconnect the attacker hosts [4], [5]. Furthermore, in recent years, the rise of IoT devices with low-cost, high-bandwidth connections has exacerbated these attacks by providing a vast pool of vulnerable devices for exploitation.