Anomaly-Based Intrusion on IoT Networks Using AIGAN-a Generative Adversarial Network

Adversarial attacks have threatened the credibility of machine learning models and cast doubts over the integrity of data. The attacks have created much harm in the fields of computer vision, and natural language processing. In this paper, we focus on the adversarial attack, in particular the poisoning attack, against the network intrusion detection system (NIDS), which is often viewed as the first line of defense against cyber threats. We develop a generative adversarial network (GAN) in AIGAN, which uses deep learning techniques to generate adversarial data and to conduct an anomaly attack on IoT networks. To evaluate the effectiveness of our generator, we measure the similarities between real and fake data using the Jaccard similarity index, in addition comparing the F1-scores from four generic algorithms: multilayer perception, logistic regression, decision tree, random forest. We contrast the performance of ten machine learning classifiers experimented on two real IoT datasets and their fake adversarial samples. Our work highlights a vulnerable side of NIDS created by machine learning when attacked with adversarial perturbation.


I. INTRODUCTION
The art of deception is used to create a different persona to trick others in order to gain confidence. As the world progresses into the fifth industrial age, scammers no longer need to show their faces to continue to deceive [1]. Digital scams of cellular text messages or email became viable options [2]. Even high-value artworks no longer require paint or brush to complete [3]. With the quick development of the metaverse [4], the virtual world on top of our physical world, each of us has a virtual identity that's connected to our real-world identity. However, our virtual identities face the threat of adversarial attacks [5]. In recent years, adversaries have the ability to synthetically generate falsified data in the form of images [6] and voices [7]. Human beings can no The associate editor coordinating the review of this manuscript and approving it for publication was Rajeeb Dey . longer differentiate real and fake using eyes, one of the typical five senses humans rely so heavily on to navigate the physical world. Adversarial attacks share a common ground in adversarial sampling, which is the process of adding noise to input data to cause perturbations in neural networks (NN) to misclassify [8]. Lately, the techniques have been shifted and applied to network data [9], [10], [11]. Adversarial samples undermine data integrity, a foundation any machine learning model relies heavily on [12].
Attackers can approximate a white-box attack by using the notion of ''transferability,'' which means that an input designed to confuse a certain Machine learning (ML) model is able to trigger a similar behavior within a different model [13]. In this work, we model a white-box attack by evaluating our examples against a variety of ML models, thus showing performance over a wide range of possible intrusion detection systems.
Cybersecurity is crucial in nearly all aspects of our lives, including social, economic, and political systems. According to a report from SlashNext, 1 phishing attacks alone are projected to increase by 61% in 2022 significantly. This alarming trend underscores the need for effective measures to counter such threats. AIGAN's capacity to generate realistic synthetic data addresses the challenge of obtaining labeled data for training in this domain, thus enhancing the quality and diversity of available datasets. By training intrusion detection models using AIGAN-generated adversarial instances, the system can develop resilience and effectively detect novel intrusion techniques.
Network Intrusion Detection System (NIDS) monitors network traffic to detect abnormal activities, such as attacks against hosts or servers. Trained on normal and attack traffic, ML classifiers offer the benefit of spotting novel differences in network traffic, and classification outcomes lead to further preventive or mitigation actions. Our previous works have demonstrated the effectiveness of our self-designed deep learning models on anomaly detection [14], [15], [16] and attack detection [17], [18], [19]. However, an increasing trend lately has been applying adversarial attacks on network datasets [20], [21]. In this work, we study the deception of adversarial ML by the following two steps. First, we generate fake data from our existing lab datasets [15], [17] using a deep learning (DL) technique in generative adversarial network (GAN) [22]. Next, we leverage the information at hand to conduct an adversarial attack on our NIDS [15], [17]. We want to demonstrate that our ML models are not immune from perturbations, resulting from different adversarial attacks. Although GAN is often applied as a mitigation solution while presented as a problem [23], [24], [25], for this work, we mainly focus on the sample generation ability of GAN [26]. The results of this work can assist organizations with a vulnerability analysis of their networks and raise awareness of adversarial attacks on cybersecurity. Given our results, a cyber adversary with some insight on a target network NIDS could effectively attack that network with the most effective perturbation algorithm.
Adversarial machine learning can be categorized into four types of attacks, poisoning, evasion, extraction, and inference, as classified by the National Institute of Standards and Technology (NIST) [27].
Poisoning is adversarial contamination of training data [28]. ML systems can be re-trained using data collected during operations. An example of this is when NIDS is frequently retrained with new data. However, an attacker can tamper with this data by inserting malicious samples that can cause problems during retraining [29].
Evasion attacks [30], [31] are the most prevalent type of attack. Spammers and hackers frequently try to avoid being detected by obscuring the content of their spam emails and malware. They conceal footprints to evade identification and appear legitimate without manipulating the 1 https://slashnext.com/the-state-of-phishing-2022/ training data. An illustration of this technique is imagebased spam, where the spam message is concealed within an attached image to prevent textual analysis by anti-spam filters.
Extraction [32], [33], or model stealing, involves an adversary probing a black box ML system in order to either reconstruct the model or extract the data it was trained on. This can lead to issues when either the training data or the model itself is sensitive and confidential. For example, model stealing could be used to extract a proprietary stock trading model which the adversary could then use for their own financial benefit.
Inference attacks [34] leverage overgeneralization of training data, a common weakness of supervised ML models, to identify data used during model training. Security concerns are raised for models trained on sensitive data such as medical records and personally identifiable information, as attackers can carry out their attacks without requiring knowledge or access to the target model's parameters. The increasing popularity of transfer learning and the availability of state-of-the-art ML models in the public domain enables tech companies to develop models based on public ones, providing attackers with easy access to information about the model's structure and type. However, membership inference relies heavily on overfitting resulting from poor ML practices, meaning a model that generalizes well to the real distribution of data should theoretically be more secure to membership inference attacks.
The attack we conduct is a poisoning attack. A poisoning attack happens when the adversary has access to either the data input or the detection models of the NIDS. Given the access, the adversary has the ability to inject bad data into the model's training pool, and hence the model learns something it shouldn't. For example, Steinhardt [35] reported that, even under strong defenses, a 3% training data set poisoning leads to an 11% drop in accuracy. An article [36] even goes as far as to say that poisoning attack is the biggest threat to AI models. The poison attack takes place before the model is trained. They aim at modifying a part of the data used for training purposes to corrupt the final model. Unlike a blackbox attack, in which the adversary has neither information about the learning algorithms nor the training dataset, we proceed with the assumption that the adversary doesn't have access to the model algorithms but can modify the training data, otherwise known as a white-box setting.
The contributions of this work are: • Authorized licensed use limited to the terms of the applicable license agreement with IEEE. Restrictions apply.
The rest of the sections are organized as such: Following the premises of the taxonomy of our work from the current section, Section II reviews the state-of-the-art in adversarial ML. Section III describes the methodology used for generating our adversarial samples. We elaborate further on the selected datasets, the intuition, and design of our GAN model, the experimental setup, as well as the metrics used to benchmark the perturbation effectiveness. Section IV presents the results based on our application of multiple model-based IDS and contrasts their predictive results on the original dataset and the generated samples. Additionally, we examine our findings. Section V concludes our work by summarizing our findings, limitations, challenges and constitute future research directions.

II. RELATED WORKS
The preeminence of adversarial attacks has seen increasing notice in the recent literature [37], [38], [39], [40], [41]. The methods such as the Monte Carlo (MC) simulation method [42], generic algorithms [43], swarm particles [44], variational autoencoder [45], GAN [22] -to create synthetic data, or adversarial samples, from existing data have recently been investigated on NIDS [46], [47], [48]. The common goal is adding noise to data and creating spatial differences between the real dataset and fake dataset, making it difficult for classifiers to recognize the difference. Our previous works [15], [17] have discussed the effectiveness of our deep learning-based models in intrusion detection. In this paper, we mainly focus on the literature overview of the development of GAN through the ages and describe how generating adversarial samples using GAN has been applied in cybersecurity.
As aforementioned, GANs have been applied in anomaly detection [49], [50], [51], [52], [53], as well as data augmentation [54], [55], [56], [57], [58]. To provide a solid foundation for adversarial training [59], which is the process of retraining models once the training sets are injected with new adversarial samples, data augmentation must not be overlooked. The most applied areas when GANs are used for data augmentation are in health [46], and several studies recently started exploring the technique as an alternative for data imbalance [60], [61]. Yang et al. [62] demonstrate that generative method can speed up the poisoned data generation rate by up to 239.38x compared to the direct gradient method. Miao et al. [63] used crowdsourcing to demonstrate the effectiveness of generative poisoning attack especially when the data points/datasets are huge/large. Arora et al. [64] presented a systematic literature review of GANs applications in the cybersecurity domain, including an analysis of specific extended GAN frameworks and currently used stable cybersecurity datasets. Their work compared how security professionals are employing GANs to produce amazing results in fields, such as Intrusion Detection, Steganography, Password Cracking, and Anomaly Generation.
The work of Zenati et al. [65] uses GANs along with an encoding network to develop a unique anomaly score. This work assists with creating real-time detectors but does not contribute to the field of adversarial ML. Andresini et al. [66] created the MAGNETO system which uses GANs to create training data for IDS. They treat netflow data as 2D images and then use GANs to create training data to resolve the issue of unbalanced training sets. Even though both works gain high attention by attempting to apply transfer learning with GAN from computer vision to cybersecurity, generating adversarial examples is not their primary purpose.
In [67] GANs are used for data augmentation of network traffic data represented in 1D arrays of flow features, while a Random Forest is subsequently trained with the augmented training set. A similar approach is illustrated in [68], with GANs used for data augmentation, while Logistic Regression (LR), support vector machines (SVM) or Feed-forward Deep Neural Networks are trained for the classification. Shin et al. [69] propose the use of Sequence Generative approaches (SeqGAN and Seq2Seq) to generate new data in a sequence of network flows. Finally, Wang et al. [70] use random feature nullification to build an adversary-resistant deep learning model in malware detection.
The common trait of the research mentioned above is that they use GANs to generate rare adversarial samples and achieve a balanced condition before training the classification model. However, to the best of our knowledge, the performances are not as well as what GANs achieved with image data [71].
Piplai et al. [72] conducted an adversarial poisoning attack, which titled NAttack! Adversarial Attacks, by applying Fast Sign Gradient Method (FSGM) [73] to effectively perturb the 'attack' samples so that the classifier trained in the previous step, is forced to classify them as 'non-attack'. The attack was carried out on a dataset from the IEEE BigData 2019 Cup:Suspicious Network Event Recognition challenge [74].
Hu and Tan [75] propose a GAN-based algorithm named MalGAN to generate adversarial malware examples which are able to bypass black-box machine learning detection models. Drawing ideas from genetic programming, Xu et al. [76] propose a generic method to evaluate the robustness of classifiers under attack. Their key idea is to stochastically manipulate a malicious sample to find a variant that preserves the malicious behavior but is classified as benign by the classifier. More recently, Alzantot et al. [77] introduced GenAttack, a gradient-free optimization technique that uses genetic algorithms for synthesizing adversarial examples in the black-box setting. Mosli et al. [78] created Adversari-alPSO, a black-box attack that uses fewer queries to create adversarial examples with high success rates. Adversari-alPSO is based on particle swarm optimization [79] and is flexible in balancing the number of queries submitted to the target compared to the quality of imperceptible adversarial examples. However, they do not consider the constraints on their data fields when crafting their examples and thus risk causing the attacks to fail. The authors in [80] use a GAN to generate adversarial values for use in features input into their particle swarm optimizer that generates adversarial examples in IDS environments. While they acknowledge that features have constraints in this domain, they only modify a small number of features and modify features such as the Transport level protocol field which will cause the traffic to fail. Our work does not exclude any features from the original datasets. Additionally, their PSO heuristic attempts to minimize the distance between the original feature and the adversarial feature while our work does not attempt to do this. Devine and Bastian [81] used a ML approach for robust malware classification that integrates an MC simulation for adversarial perturbation with meta-learning using a stacked ensemblebased methodology.

III. METHODOLOGY
In this section, we give a technical description of the features used in the datasets. We then explain the details of the techniques employed for adversarial example generation. This leads to the layout of our computational setting. The section concluded with the listing of classifiers used for testing.

A. DATASET
We used two datasets to assist researchers in detecting cyber anomalies and to fill the void of publicly available IoT datasets [15], [17].
The first dataset is titled the Center for Cyber Defense (CCD) IoT Network Intrusion Dataset V1 (CCD-INID-V1), used in our previous work [15]. On top of creating the dataset, we propose a hybrid lightweight form of IDS-an embedded model (EM) for feature selection and a convolutional neural network (CNN) for attack detection and classification. The proposed method has two models: (a) RCNN: Random Forest (RF) is combined with CNN and (b) XCNN: eXtreme Gradient Boosting (XGBoost) is combined with CNN. RF and XGBoost are the embedded models to reduce less impactful features. We attempt anomaly (binary) classifications and attack-based (multiclass) classifications on CCD-INID-V1 to explore the effectiveness of these learning-based security models.
The second dataset is titled as the CCD-IDSv1. The dataset was created in an OpenStack environment as described in our previous work [17]. The dataset was labeled based on network flows as they were identified using our filter rules. In addition to creating the dataset, we developed two DL-based ensemble models: Ensemble-CNN-10 and Ensemble-CNN-LSTM. Ensemble-CNN-10 combines 10 CNN models developed from 10-fold crossvalidation, whereas Ensemble-CNN-LSTM combines base CNN and LSTM models. The work highlights feature importance for both anomaly detection and multi-attack classification.

1) CCD-INID-V1
This dataset was generated using a hybrid model, which combines physical devices with network virtualization functions. The data originate from our Raspberry Pis. The dataset contains 83 features generated from NFStream [82], which identifies bidirectional flows from netflows. The steps are illustrated in Figure 1. The generation architecture is illustrated in Figure 2. The dataset carries five attack types that were prevalent at the time of creation. The five attacks are ARP Poisoning, ARP DoS, UDP Flood, Hydra Bruteforce and SlowLoris. The dataset was created to resemble the characteristics of IoT devices as they transmit data to cloud servers. Some of the features are categorical whereas some are continuous. The full listing with description of the features can be found in [15]. The dataset contains two variations: anomaly and multi-classed. In the anomaly dataset, attack is labeled as 0 and benign labeled as 1.

2) CCD-IDS-V1
This dataset was created in a fully virtualized environment. The network architecture was created using Open-Stack. OpenStack is a cloud operating system that controls large pools of computing, storage, and networking resources throughout a data center, all managed and provisioned through APIs with common authentication mechanisms.
The network environment, as shown in Figure 3, is implemented on OpenStack. Two internal networks, internal_1 and internal_2, are created. Five instances of Operating System  are created for each internal network, respectively. For this research work, Linux environments, Kali and Ubuntu, are used. Kali Linux is primarily used as a penetration testing environment that contains different attacks by default. Five different Kali systems are used to attack five different Ubuntu systems in parallel. The setting enables scalability for the number of connected devices as well as the attack diversity. The current attacks are carried out by Kali systems on Ubuntu systems at random time intervals. Every Kali system will infiltrate the Ubuntu system.
To extract the features from the raw PCAP files, first we convert the files into the Argus compatible format. Argus is a data network transaction auditing tool that categorizes and tracks network packets that match the libpcap filter expression into a protocol-specific network flow transaction model. Argus reports on the transactions that it discovers, as periodic network flow data, that are suitable for historic and near real-time processing for forensics, trending, and alarm/alerting. In this research, 25 features/attributes, shown in Table 2 of [17], are extracted from both malicious and normal traffic. These attributes consist of network flow information, including their statistical properties as well. The final CCD_IDSv1 dataset is in CSV format for evaluation. The dataset is labeled in two different ways: for anomaly detection and threat or multi-attack classification. Anomaly detection is binary classification, so the dataset is labeled into two classes: normal and attack. For threat classification, each different attack is labeled, including normal usage for multiclass classification.

B. ADVERSARIAL SAMPLES
Based on the datasets with real data, CCD-INID-V1 and CCD-IDSv1, created in [15], we inject noise using GAN to generate two synthetic datasets. The generation process consists of two parts -adversarial training, adversarial data generation.
GAN is typically composed of two deep neural network models: discriminator (A), and generator (B). The discriminator attempts to discern if its inputs are from the genuine data set or from the adversarial data set. The generator's task is to learn from the discriminator's output and thus train so that its output may deceive the discriminator.
The training of GANs is based on a zero-sum or minimax game with two players, each one (A and B) trying to maximize its own benefits. The game converges when both players reach a point where changing their actions (updating the weights of neural networks) does not bring more benefits (or the loss functions for A and B cannot be further minimized). This point is the Nash equilibrium for the following equation: • A(xi) is the discriminator's estimate of the probability that real data instance xi is real.
• B(zi) is the generator's output when given noise zi.
• A(B(zi)) is the discriminator's estimate of the probability that a fake instance is real.
• The formula is derived from cross-entropy between real and generated distributions.
This equation shows that B tries to minimize the loss function while A tries to maximize it.
The Generator is a neural network model responsible for generating realistic samples from the target domain. The input for the generator model is a vector randomly sampled from a uniform or Gaussian distribution. This vector is used as a starting point for the G model to generate synthetic data   in the problem domain. This random vector represents a compressed version of features of the outputs referred to as latent features or a latent vector. In fact, during the training process, the Generator converts this random vector to meaningful data points. In this way, each new random vector drawn from the latent space is converted to a new output in the problem domain.
Adversarial data generation does not occur before adversarial training. But the process is sequential yet iterative as it is carried out over and over before reaching an acceptable stage. The process typically consists of five steps.
Step 1 is to define the scope of research in order to select the right architecture for GAN. For this work, we apply our own AIGAN to generate IoT data in order to perform an anomaly intrusion attack on NIDS. AIGAN stands for an anomaly-based intrusion using generative adversarial network. The architecture is exhibited in Figure 5.
Step 2 is training the generator. The summary that describes the learning layers of AIGAN's generator can be found in Figure 6. When the generator is trained, the discriminator is idle. During the generator training through any random noise as input, the generator tries to transform the input into meaningful data. The generator gets random noise using latent dimensions. Intertwined with the next step, which is training the discriminator, the output from generator is passed into the discriminator to be classified as either real or fake. As the discriminator loss is calculated, backpropagation is performed on both the discriminator and the generator to calculate gradients. The gradients are used to update generator weights.
Step 3 is used for training the discriminator. Figure 7 demonstrates the composition of AIGAN's discriminator. When the discriminator is trained, the generator is idle. The discriminator is trained on the real dataset. Initially started off with only a forward path, no backpropagation needed with the first training of the discriminator. The discriminator classifies both real and fake data. The calculated loss helps improve its performance and penalizes it when it misclassifies real as fake or vice-versa. Then the weights of the discriminator are updated through discriminator loss.
In Step 4, the discriminator is trained on fake data. The samples which are generated by the generator will are passed into the discriminator. As the discriminator attempts to label real and fake, the feedback is presented to the generator repeatedly.
The discriminator is trained using the complete adversarial vectors from the generator along with benign vectors from the real data set. These vectors are then classified by the discriminator and previous classifiers as benign or malicious. The outputs of the two classifiers are compared in the following way: If both classify an input as malicious, then the feedback to the discriminator is labeled malicious. Otherwise, the classification is benign. In the next step, the generator is once again trained based on the feedback given by the discriminator and tries to improve performance.
These steps are repeated iteratively until the discriminator no longer can be fooled by the generator.

C. EXPERIMNT SETUP
The overall accuracy performance of the proposed methodology is measured by analyzing the F1-score of the intrusion detection models learned. This is the harmonic mean of Pre-cision and Recall, where Precision measures the ability of an intrusion detection system to identify only the attacks, while Recall can be thought of as the system's ability to find all the attacks. The higher the F1-score, the better the balance between Precision and Recall achieved by the algorithm. On the contrary, the F1-score is not so high when one measure is improved at the expense of the other. In addition, we consider Accuracy (that is measured in the evaluation of various competitors). This is the ratio of flows correctly labeled on all flows tested. All these metrics are computed on the testing set of the considered dataset.

1) ENVIRONMENT
The efficiency performance is evaluated with the computation time spent training the intrusion detection model. The computation time is collected on a Windows machine with an Intel(R) Core (TM) i7-10870H CPU @ 2.21GHz and 64 GB RAM. All the experiments are executed on a single GeForce RTX 3080. The duration of computation is expressed in units of minutes and seconds.

2) EVALUATIONS
We want to assess the effectiveness of adversarial malware samples against the state-of-the-arts ML models. During adversarial training, we look at the progression between generator loss versus discriminator loss. We compare the F1 scores of the models when applied on the original real datasets versus the F1 scores when applied on the synthetic datasets. The NIDS classifiers include random forest (RF), decision trees (DT), logistic regression (LR), k nearest neighbor (KNN), Naïve Bayes (NB), support vector machines (SVM), AdaBoost, Gradient Boosting (GBoost), Convolutional Neural Network (CNN) [17], RCNN (RF+CNN) [15].
Besides comparing the performance of the ten listed classifiers, we provide comparative analysis with the assistance of a toolkit named Table Evaluator [83]. TableEvaluator is a library to evaluate how similar a synthesized dataset is to a real data. In other words, it tries to give an indication into how real your fake data is. The toolkit applies four classifiers to compare the F1 scores of real and fake data and calculates the similarity score using Jaccard.
The F1 score is a machine learning evaluation metric that measures a model's accuracy. It is calculated by combining the precision and recall scores of the model. The precision measures the proportion of correctly predicted positive instances out of all instances predicted as positive. In contrast, recall measures the proportion of correctly predicted positive instances out of all actual positive instances in the dataset. The Jaccard similarity index, sometimes referred to as the Jaccard similarity coefficient, compares members of two sets to see which members are shared and which are distinct. It's a measure of similarity for the two sets of data, with a range from 0 to 1. The higher the percentage, the more similar the two populations. The formula of Jaccard is shown 91122 VOLUME 11, 2023 Authorized licensed use limited to the terms of the applicable license agreement with IEEE. Restrictions apply.   in Figure 9. The four classifiers are DT, LR, multi-layer perceptron (MLP), and RF.
Besides the above metrics, we show the means and standard deviations between real and fake data. For each dataset, we show the cumulative sums and distributions to certain highlighted features. We also demonstrate the correlation matrix plots and principal component analysis (PCA) to both datasets.

3) PARAMETERS
We are performing the adversarial attack on our NIDS, which consists of eight generic machine learning algorithms and two pre-trained deep learning models.
The two pre-trained deep learning models are taken from our previous works [15], [17]. The structure and hyperparameters for our pre-trained CNN are shown in Figure 8. The summary to RCNN is as follows:

A. GENERAL RESULTS
Since the attack types from both datasets overlap, we can see how the adversarial poisoning attack performed when seen in different situations. The results also provide us with insights on potential weaknesses to the datasets for future improvements. The subsequent discussion of the results indicates where useful similarities are found between real and fake data. It should be noted that no features were dropped for the experiments in order to provide the most realistic results.

B. CCD-INID-V1
In this section we demonstrate the findings for the dataset CCD-INID-V1.   In the table shown in Figure 10, the row index contains the name of the estimator and the name of the dataset. The second and third columns provide the F1-scores on show how they perform when predicting real and fake data. The last column shows the similarity score according to the Jaccard similarity index. Using row 2 as an example, the DT estimator is applied on the real data. The estimator correctly classifies data as real with a F1-score of 1.00 and classifies data as fake with a 0.50 F1-score. The similarity is consistently approximately around 0.4.
We select a few features that provide us with the best insights to the comparisons between real and fake data out of the 83 features. By looking at the cumulative sums of the selected features, we observe the generator successfully learn certain characteristics of real features, particularly in Figure 11. - Figure 16. The cumulative sums, also known as running totals, are the data sum's progression through time. It displays the total contribution of a certain measure against time. The real data is shown in blue, and the fake data is shown in orange. The more overlap between the two data, the harder to differentiate between real and fake.
In figures 17, 18 and 19, the distributions of selective features are shown. Orange resembles fake data and blue is real data in the histograms.   By examining the correlation between real and fake data in figures 20, 21, and 22, we observe that marginal differences exist between real and fake data.     But in Figure 21, the features from the fake dataset are highly correlated. The redder shown means higher positive correlation whereas the bluer shown means higher negative correlation. Figure 22 shows that real and fake has an evenly spread. The more redness means more difference whereas the whiteness means no difference. Figures 20,21,22 illustrate that for human's set of eyes, it is visually hard to clearly identify a pattern of difference between the real data and the created synthetic fake data. This confusion is exactly how AIGAN is affecting decision making of the classifiers.
In Figure 23, the fake data is more evenly spread across the board compared to the real data.
In Table 1, we observe the perturbations caused by the fake data generated with AIGAN. For most classifiers, the F1scores appear to suffer a drop in scoring from real data to fake data. Our pretrained RCNN model suffered a 11% drop and NB suffered a 22% drop.
And finally, we see the loss progressions between the competition of generator and discriminator in Figure 25. We can VOLUME 11, 2023 91125 Authorized licensed use limited to the terms of the applicable license agreement with IEEE. Restrictions apply.  see the generator attempts in the first 50 epochs to deceive the discriminator until it cannot do so. Discriminator is able to consistently handle the generator's attempts.

C. CCD-IDSV1
Similar to Figure 10, using the Jaccard similarity index, the similarity score is consistently approximately around 0.4. The similarity score achieved 1.00 when applied MLP classifier on real dataset and 0.9231 when applied LR on real dataset.
Just as the previous dataset, we selected several features out of the 23 features from CCD-IDSV1 to provide insights.
Looking at Figures 26 to 33, we can see AIGAN was able to create the replicates to these features at certain points.   By examining the correlation between real and fake data in figures 34, 35, and 36, we observe extensive differences exist between real and fake data compared to CCD-INID-V1. 91126 VOLUME 11, 2023 Authorized licensed use limited to the terms of the applicable license agreement with IEEE. Restrictions apply.   In Figure 34, we identify higher positive correlations among several features by spotting the clear redness. Figure 35 demonstrates that features have fewer positive correlations but have slightly more negative correlations. Combine the patterns we see in Figure 34 and Figure 35; Figure 36 confirms that differences between real and fake data is more noticeable since more bright red is shown and less white blocks exist in the figure.   Identical to what was shown in Figure 23, the PCA spread for fake data is more random than the straight-line shape for real data.    In Table 2, just as it was the case for CCD-INID-V1, F1-scores prove perturbations exist under the influence of AIGAN. DT has a drop of 13% while NB has a drop of 25%  in F1-score. Our pretrained CNN model suffered the most with a drop of 55%.
Just as the case of CCD-INID-V1, AIGAN's discriminator was able to easily beat the generator in adversarial training. The results demonstrated that AIGAN was able to incur perturbations in ML classifiers when applied on the CCD-INID-V1 dataset and the CCD-IDSV1 dataset. AIGAN impacted the most against the pretrained CNN models, which was proven to provide high detection rates without the threat of adversarial attack. However, AIGAN's generator did not match up well with the current discriminator.

V. CONCLUSION
In this research, we seek to assess the effectiveness of adversarial training, in particular using GAN, towards the generic 91128 VOLUME 11, 2023 Authorized licensed use limited to the terms of the applicable license agreement with IEEE. Restrictions apply.   classification models as well as our self-generated models. Our work highlights the vulnerability of ML-based NIDS in the face of adversarial perturbation.
Even though GAN has been used on some publicly available datasets such as NSL-KDD, and USWF-15, we extend the testing on our own datasets. Not only do we want to see how effective the attack can be, but also to seek insights into possible future directions. We evaluated the effectiveness of proposed methodologies using two benchmark IoT datasets, namely the CCD-INID-V1, and CCD-IDSV1. The experimental results prove the viability of our proposed methodology. The fake network data generated by AIGAN was able to cause perturbations not only in classical generic algorithms but also in our pretrained CNN models. However, our discriminator was able to identify the real and fake data from both datasets.
There are a few limitations in this study. First, the dataset used in this study may have inherent biases, which could impact the performance and accuracy of the proposed method. Awareness of these biases and their potential impact is important. Second, the scalability of the method to larger and more complex IoT networks should be considered. While promising results may be observed on a smaller scale, the performance and efficiency may vary when applied to larger networks. Further investigation is needed to assess scalability.
The challenges associated with the proposed method are notable. Adversarial attacks evolve and become more sophisticated, posing challenges for effectively detecting and mitigating new and unseen attack types. Continuous research and updates to the method are necessary to stay abreast of the evolving threat landscape. Furthermore, generalizing the method's effectiveness and performance across diverse IoT domains and datasets can be challenging. Adequate testing and evaluation across various scenarios and datasets are crucial to ensure its applicability in different contexts.
As future work, we plan to explore the effectiveness by looking into further existing methods, such as MAGNeTo, Conditional GAN, Wasserstein distance metric to train the networks (WGAN), particle swarm optimization, to create adversarial malware examples. When applying these techniques, we will consider the challenges faced by current GANs. One challenge is to properly balance generator and discriminator. Discriminator has to be more lenient to allow generator to create better synthetic data. GANs also lack the understanding holistic structure to the entire datasets. Injecting the element of explainable artificial intelligence techniques [84] is a potential solution to strengthen this weakness. Similar to how information is used to explain image classifications, we can explore this to identify an attack signature by putting the traffic characteristics that are most relevant for each attack family in the spotlight. To this end, we plan to extend the current investigation to signature-based classifications.
YANG LIU (Member, IEEE) is currently pursuing the Ph.D. degree with the Department of Computer Science, North Carolina Agricultural and Technical State University. He is a Graduate Research Assistant and a Teaching Assistant. His research interests include natural language processing (NLP), social media mining, public health surveillance, artificial intelligence and machine learning, and quantum computing.
KAUSHIK ROY (Senior Member, IEEE) is currently a Professor and the Interim Chair with the Department of Computer Science, North Carolina Agricultural and Technical State University (NC A&T), and the Director of the Center for Cyber Defense (CCD). He also directs the Cyber Defense and AI Laboratory. His research is funded by the National Science Foundation (NSF), the Department of Defense (DoD), and the Department of Energy (DoE). He has more than 150 publications, including 40 journal articles and a book. His research interests include cybersecurity, cyber identity, biometrics, machine learning (deep learning), data science, cyber-physical systems, and big data analytics.

XIAOHONG YUAN (Senior Member, IEEE)
is currently a Professor with the Department of Computer Science, North Carolina Agricultural and Technical State University (NC A&T). Her research has been funded by the National Security Agency, the National Centers of Academic Excellence in Cybersecurity (NCAE-C), the National Science Foundation, the Department of Energy, and the Department of Education. Her research interests include AI and machine learning, anomaly detection, software security, cyber identity, and cyber security education. She has served on the editorial board for several journals on cybersecurity.
JINSHENG XU (Senior Member, IEEE) received the B.S. degree from Nanjing University, China, the M.S. degree from Peking University, China, and the Ph.D. degree from Michigan State University. He is currently an Associate Professor with the Department of Computer Science, North Carolina Agricultural and Technical State University (NC A&T). He is participated in numerous funded projects in cyber security research and education. He teaches Python for data science, network security, data structures, advanced algorithms, and other courses. He has published many peer-reviewed research papers on the topics of cyber security. His research interests include network security, machine learning, and modeling and simulation.