Smart IoT Network Based Convolutional Recurrent Neural Network With Element-Wise Prediction System

An Intelligent Internet of Things network based on an Artificial Intelligent System, can substantially control and reduce the congestion effects in the network. In this paper, an artificial intelligent system is proposed for eliminating the congestion effects in traffic load in an Intelligent Internet of Things network based on a deep learning Convolutional Recurrent Neural Network with a modified Element-wise Attention Gate. The invisible layer of the modified Element-wise Attention Gate structure has self-feedback to increase its long short-term memory. The artificial intelligent system is implemented for next step ahead traffic estimation and clustering the network. In the proposed architecture, each sensing node is adaptive and able to change its affiliation with other clusters based on a deep learning modified Element-wise Attention Gate. The modified Element-wise Attention Gate has the ability to handle the buffer capacity in all the network, thereby enriching the Quality of Service. A deep learning modified training algorithm is proposed to learn the artificial intelligent system allowing the neurons to have greater concentration ability. The simulation results demonstrate that the Root Mean Square error is minimized by 37.14% when using modified Element-wise Attention Gate when compared with a Deep Learning Recurrent Neural Network. Also, the Quality of Service of the network is improved, for example, the network lifetime is enhanced by 12.7% more than with Deep Learning Recurrent Neural Network.


I. INTRODUCTION
THE infrastructure of Wireless Sensor Networks (WSN) is built in an ad hoc way with arranged nodes informing a Base Station (BS) about events. A WSN is incorporated with smart technologies to deliver fast Internet of Things (IoT) communications among different applications [1], [2]. A WSN with IoT in a real-world application has to convey a wide range of data and owing to the limited bandwidth, these different types of data need to be managed with different priorities in order to avoid congestion in the network [3].
Area monitoring, where multiple IoT nodes make an assessment of the data and communicate them to the gateway or to the cloud for processing, assumption, and analysis, The associate editor coordinating the review of this manuscript and approving it for publication was Xiaolong Li . is a crucial responsibility of the IoT. This communication is costly when these data are high-dimensional (e.g., videos or time-series data). IoT networks having limited bandwidth and hence, low power devices may not have the capacity of supporting frequent transmissions with high data rates. With the advent disruptive IoT involving huge amounts of different types of data, Machine Learning (ML) and Deep Learning (DL) mechanisms will play a pivotal role in bringing intelligence to these networks [4], [5]. Among other affordances, ML and DL can play an essential role in addressing the challenges of resource management in large-scale IoT networks [6]. Applying deep neural networks to IoT devices could, thus, bring about a generation of applications capable of performing complex sensing and understanding tasks for supporting a new realm of cooperation between humans and their physical environment [7].
The implementation of Convolutional Neural Networks (CNNs) in different fields has increased in recent years [8]- [10]. The main focus of this paper is to design an intelligent controller based on a deep learning CNN. Moreover, an algorithm that is more efficient than the traditional NN is identified for training the proposed system and thus, achieving high Quality of Service (QoS).

A. MOTIVATION
The most significant matter in the subject of traffic prediction in an IoT network is the amount of data stream. An increase in the number of sensors that communicate with the IoTrouters in the network leads to raising the traffic load in the sink node buffer. A major concern of researchers is the use of Artificial Neural Networks (ANNs) and ML with IoT networks. A special issue on artificial intelligence and ML in [11] outlines the AI algorithms that have been implemented in networking and communications. The proficiency of deep learning based on ANNs has been demonstrated in security, routing, traffic management and load balance in an IoT network [10], [12]- [14].
While managing the QoS, optimizing the routing process of the network is an essential requirement for enhancing network performance. Thus, it is essential to build up an intelligent routing mechanism that has the ability to learn from the past behavior of network and then, be able to adjust itself to the current behavior [15]- [17]. QoS provisioning is the most important performance metric when designing future wireless networks. In practice, the buffers are limited and thus, data buffering leads to buffer overflow and delay, both of which are important QoS considerations [18].

B. CONTRIBUTIONS
The contributions of this paper are summarized as follows: 1. We propose an artificial intelligent system based on a deep learning modified Element-wise Attention Gate (EG-CRNN) to predict the amount of packets in the network and to manage the cluster head of the sensors. The EG-CRNN has an Element-wise Attention Gate and self-feedback aimed at increasing its long short-term memory; 2. We propose a training algorithm for a deep learning EG-CRNN to update the weights of EG-CRNN and to speed up the training process in order to reach the error goal; 3. To explain the effectiveness of the architecture proposed, a comparison is made among the Artificial Intelligent System (AIS) based on EG-CRNN with others based on different structure of CNN in terms of speeding up the training process and minimizing the error.
The rest of this paper is presented as follows. Section II reviews related works, whilst section III presents the proposed system architecture and section IV presents the proposed artificial intelligent system. Then, in section V, the training algorithm of the proposed intelligent system is explained. In section VI, the evaluation setup is provided and the results are provided, with the QoS metrics being evaluated. The conclusion to the paper is presented in section VII.

II. RELATED RESEARCH WORK
Sensor-integrated IoT devices are the main handler for gathering large amounts of data. Managing these huge data sets is one of the critical challenges to be undertaken. In this section two important strategies are taken into consideration. The first one is how to select an appropriate CNN as a controller to overcome the congestion in I-IoT network, and the second one is how to find a clustering technique based on CNN rather than that one based on traditional method as in [19], [20]. The most up-to-date research concerning the deep learning CNN use in different applications is also introduced in this section.
The CNN was used to increase the link capacity and energy efficiency of the available channel state information at the base station in wireless network. The proposed architecture of CNN as in Guo et al. [21] aimed at improving the bandwidth resources. The idea of adding more layers in neural network to have deep neural network was proposed by Mukherjee et al. [13] for optimization transmit power in an industrial IoT network with a clustering approach based on a deep learning CNN technique, with which they were able to maximize the security capacity of the network and improve the QoS. The CNN can be more powerful when adding recurrent feedback to its structure. Mou et al. [22] proposed a Recurrent Convolutional Neural Network (ReCNN) structure trained to learn a joint spectral-spatial-temporal characteristic representation in a uni?ed framework for change detection in the field of image processing. This structure was shown to provide good results in multi-temporal remote sensing image analysis. Sonny et al. [23] proposed Long Term Evolution (LTE)-based Channel State Information (CSI) and CNN for smart parking solutions to minimize traf?c congestion. Belmonte-Hernández et al. [24] used deep RNN with WSN for indoor person tracking to improve the estimation of the position captured by wireless sensors. Yang et al. [25] proposed a hybrid architecture, named the Parallel Recurrent Convolutional Neural Network (PRCNN) for application in the field of mobile IoT and sensing devices. Also Zhang et al. [26] proposed a hybrid method for CNN and RNN used for estimation of the remaining useful life of prognostic health management technology, which provides better performance of this estimation than CNN. Many researchers focus on increasing the ability of NNs to have concentration effectiveness. For example, Zhang et al. [27] proposed a simple effective Element wise-Attention Gate (EleAttG), which can be easily added to an RNN block (e.g., all RNN neurons in an RNN layer), to empower the neurons to have concentration effectiveness. Bai et al. [28] proposed DL-RNN, a real-time wireless localization model, which consists of double recurrent neural networks (RNNs): the first RNN estimates the location and the second filters the location, which further improves wireless fingerprinting localization performance. In the field of improving the training algorithm of deep learning CNN, researchers have proposed some algorithms to empower the CNN to have more attentiveness capabilities [24]- [27]. VOLUME 9, 2021 Also, the clustering techniques are very effective criteria in the field of minimizing the congestion in I-IoT networks. Farsi et al. [29] introduced a Congestion-Aware Clustering and Routing (CCR) protocol to alleviate the congestion issue over the network. Their protocol aimed at satisfying the QoS requirements of increasing the network lifetime and raising the number of packets sent. The bandwidth allocation method is particularly effective in the overloaded network scenario, where the higher-priority traffic interferes with the other applications.
Mukherjee et al. [30] presented a Multi-Input Multi-Output (MIMO) technique model in WSNs, which addresses the cluster head recognition issue for MIMO sensor networks by using Back Propagation Neural Network (BPNN). The proposed model showed a better performance in terms of minimizing the energy consumption, the error rate, and the computation time.
In this paper, we proposed an EG-CRNN structure which combines the advantages of DL-RNN structure and the EleAttG structure in order to improve the efficiency of the I-IoT network. Thus, a comparison is carried out among the proposed structure and those up-to-date ones proposed in [23] and [25]. We utilized some features of their structures to serve our own application. Although the structures of AI proposed in [23] and [25] have different applications from our own application, the idea is how to find the best AI structure in terms of high speed of training with minim error and has the ability to predict the traffic flow in IoT network. Fig.1 shows the architecture introduced in this paper. The evolution in the science of the deep learning, CNN and communications means that these technologies have now been adopted in a wide range of life aspects. Our target is the application of an Intelligent Internet of Things (I-IoT) network in the field of smart healthcare post-COVID-19. The proposed network, as shown in Fig.1(a), consists of number of I-IoT monitoring nodes classified as a number of clusters, mobile data gathering and an IoT hospital cloud network. One solution for increasing network scalability and prolonging the life of the network is to use a mobile sink node [31]. Each cluster can be manage itself based on a deep learning EG-CRNN, as shown in Fig.1(b). The proposed architecture incorporates an IoT patient monitoring area. This area is specified as the number of sensing nodes that are categorized according to their actions into three kinds: Cluster Head (CH), effective node and quiet node. Effective member nodes transfer their data to a CH, which in turn, passes on accumulated data to the mobile data gathering node as an IoT Gateway (GW). Practically, the GW links the sensors using a bidirectional single hop connection through the Internet and the GW can connect to the Internet by regional routers with firewalls [32].

III. SYSTEM ARCHITECTURE
The arriving data flow from the buffer of the CHs through a count of active sensors is directed to the hospital cloud network.
The proposed EG-CRNN, which has a distributed decision making structure, is used to process the gathered data and to analysis them. The parameters that are used in the selection of the best CH are: remaining energy of the IoT nodes, their density and the remaining space of the buffer capacity.
The density of one node refers to the amount of adjoining nodes accumulated in a specific range place.

IV. THE PROPOSED ARTIFICIAL INTELLIGENT SYSTEM
The I-IoT network is formed in a homogeneous network, as illustrated in Fig. 1. The function of the sensors is to recognize the data from distinct medical devices with distinct kinds of traffic. The periodic data are gathered from an IoT platform, (e.g., the blood pressure of the patient, glucose level, temperature or heart beat).
In this situation, the IoT can dynamically gather patient data to catalyze protective care, diagnostics etc. and to evaluate remediation outcomes. In Fig.1(a), the hospital cloud network consists of a number of routers, which depends on the considered number of IoT devices. Each router and CH has its First-In First-Out (FIFO) buffer with a pre-known capacity.
We have proposed an intelligent controller in the network, as shown in Fig.1(b), which is based on the EG-CRNN methodology. This controller has two tasks which are explained as follows.

1) EG-CRNN PREDICTION CONTROLLER
The proposed EG-CRNN architecture has one input layer, four CRNN layers, as depicted in Fig.1(b), a Rectified Linear Unit (ReLU) layer and an output layer. All the EG-CRNN layers, except for the output layer, have the ReUL activation function, as f (net) = max(0, net). The ReUL function delivers faster than sigmoid function during the gradient decent training algorithm. The EleAttG element proposed by [27] is modified in the proposed structure, as an Element-Wise Attention Gate (EG), which is used in the RNN to empower it to have concentration effectiveness. Fig. 2 shows the structure of the RNN with the modified EleAttG and accordingly, we call this structure EG-CRNN. The RNN has four layers, these being the input layer, the two invisible layers with the self-feedback α and the output layer. Each node in the invisible layer has an Element-Wise Attention Gate. The activation function of the output layer is linear function as f (net) = net.
The present input I t and the past invisible states h t−1 are used to consider the levels necessary for each element of the input. g t is the response of the EG. The vector g t has the same dimension as the input vector I t and is computed as: where, denotes the element wise product and is the activation function. W Ig and W hg are the weight matrices. When the response of gate function is near to zero, the invisible layer state h t with its self-feedback is prompted to neglect the previous invisible state and shifts with the current input individually. Fig.3 shows the internal structure of EG, with the computations of the EG block being as follows: where, W Ir , W hr ,W IZ , W hZ , W Ih and W hh are the weight matrices in the internal element gate structure. The EG-CRNN has two main components: features extraction and classification. The features extraction of the traffic flows, like the packet generation rate and the lengths of the packet queues, is constructed of convolutional layer, which is the input to EG-CRNN. The classification component represents the output of EG-CRNN layer. Two binary values are collected from path mixtures as output. These are (1,0) where congestion occurs, and (0,1) where it does not. Those path mixtures that will not lead to congestion are chosen and the EG-CRNNs will be periodically updated when they are being used to select this path mixture. The traffic flow of every CH is recorded by the CH itself, which then sends the data to the mobile data gathering that is forwarded to the IoT GW. The EG-CRNN controller, as shown in Fig.1(b), uses the data such that the traffic patterns of all CHs will be organized in a matrix and then be utilized as EG-CRNNs input to select the path mixture for the period ahead of time. Fig.4 illustrates the proposed AIS, where the error is the difference between the target and real traffic.
To illustrate the mechanism of the proposed system, we assume that we have an IoT platform to be controlled, as shown in Fig. 4, where the traffic flow equation is computed as: where, tf (t) is the traffic flow at time t, T is the sampling period, u(t) is the control effect at time t and sat[·] is the saturation function. The nonlinear function,ff (·), represents the real traffic. This is considered to be undefined and it is a function of traffic input, buffer size, and free service capacity at the active CH nodes. The control effect at time t is computed as: (5) where, k P is the proportional gain, k V is the coefficient of the integral in the Proportional Integral Derivative (PID) controller, used to improve the accuracy of the network and to overcome the steady state error as well as maintaining the stability of the network during the learning mechanism. k D is the derivative coefficient in the PID controller used to reduce the damping ratio of the traffic. Combining the PID controller with a Deep Neural Network (DNN) accelerates the training process of the latter, which will enhance the controllability of the network [33]. Thef (tf (t)) is the predicted traffic and the tfT is the target traffic.
The minimum rate MR of the IoT node, is calculated as: where, Q C is the buffer size of (C) CH node with the rate R C . The total network utilization ψ is maximized as in equation: The traffic flow at the next time interval is described by: where γ is the count of the effective CH.

2) EG-CRNN CLUSTERING ASSIGNMENT
The other task of the AIS is to choose the best CH to sustains traffic. The I-IoT network is handled depending on the deep learning EG-CRNN. The EG-CRNN here updates its weights in an online mode. The features extraction pertains to the parameters referred to in Section III as input, while the classification for generating the output is the logical value, where logic 0 is interpreted as cluster affiliation and logic 1 is a CH.

V. EG-CRNN TRAINING ALGORITHM
The training algorithm that is used to learn the EG-CRNN is described in this section. The proposed algorithm is based on the extension of the negative gradient descent algorithm for minimizing the error between the estimated and real traffic. The input to the EG-CRNN is allocated to the traffic flow aggregation rate tf (t), i.e., the packets number reaching the mobile data gathering node from the network. There are two phases for the training algorithm in EG-CRNN: the offline and online phases. The EG-CRNN trains offline to determine the size of the buffer of IoT platform and then it trains online to identify the traffic flow. In the online training phase, the weights of EG-CRNN are updated to the new environment. During the offline training, the EG-CRNN adjusts its weight through the training process depending on random traffic as input to the EG-CRNN, which is generated from the expected traffic that the network can achieve depending on the past behavior of the network. At first, the weights are initiated arbitrarily between [-1,+1] and then, after implementing rounds of training, the weight values are updated more accurately with adaptive learning rate η. The weight of each connection is influence by the response function . The response function (g t ) is defined as: For simplicity, we define the function (g t ) as y h and (Z t ) as y i . The error E represents the difference between the target and estimated traffic, being calculated as: The weights of the invisible and output layers will be updated as stated in the following equations: where, And the Delta functions δ j and δ i are computed as: where, and Table 1 describes the symbols that are used in the above equations. The flowchart of the proposed AIS is shown in Fig. 5. The initial values of the parameters are chosen by trial and error, then being updated through the training algorithm. EG-CRNN is adaptive according to the traffic dynamics and the performance of the IoT active platform, such that the EG-CRNN controller keeps harmony between the buffer sizes and traffic flow of the network. In fact, adding EG to the CRNN is influential in enhancing many CRNN based learning actions.

VI. RESULTS AND EVALUATION
We consider scenario with 150 IoT sensors that are placed randomly in a square area of (200 × 200) meters. The application area is shown in Fig. 6, where the mobile data gathering node can move to the center site among effective clusters according to the equation: where X and Y are the positions of C effective CHs. This mechanism guarantees that the mobile data gathering node is fairly close to data IoT devices. In general, the mobile data gathering node can enhance the QoS [34], [35]. The IoT devices are generated traffic at the beginning of each scheduling period, with the IoT generating low to high flow. The EG-CRNN contributes to minimizing the congestion level regarding the occupied percentage of packets in the affiliate queue buffer. So, if this percentage overrides a threshold, the CHs are categorized as congestion. The threshold level VOLUME 9, 2021  here, is set at 95% of the FIFO buffer capacity and it is chosen based on empirical valuation.
The emulation is run with the MININET emulator and Python programming language using the parameters explained in table 2.
The following suppositions are utilized for the network: 1. The overall waiting time of the packet in the FIFO buffer represents the amount of the Round Trip (RT) communication delay in the links and the chaining processing delay in the cloud; 2. Static flow is generated by all active IoT nodes per unit of time; 3. The link between the mobile data gathering, CH, and its nodes constitutes the wireless links of a bidirectional single hop; 4. IoT devices can establish their mode as reported by the CH buffer space and its density. Fig. 7 shows the minimization of the Root Mean Square (RMS) error during the offline training phase, which is computed as: In Fig. 7 a comparison among the proposed system based on EG-CRNN, the system based on CNN and that based on a Deep Recurrent Neural Network (DRNN) without adding element wise gate is made. The structure of CNN has one input layer, four Feedforward Neural Network (FFNN) layer, a (ReLU) layer and an output layer. While, the structure of DRNN is the same as CNN but instead of FFNN layer there is RNN layer. It is evident from the figure that EG-CRNN can reach the error goal (at 22 epochs) faster than DRNN and CNN, which is because the element wise attention gate and the self-feedback enhances the training process, thus speeding up the network in reaching the error goal. In fact, the network does not need as much information in the training process as CNN. The EG-CRNN can minimize the RMS error by 56% as compared with CNN and by 37.14% as compared with DRNN. Based on the results obtained from Fig. 7, we implement the I-IoT network with the proposed EG-CRNN, and DRNN, because the latter is better than CNN in terms of minimizing the RMS error. Energy consumption of I-IoT network with respect to total number of IoT nodes is shown in Fig. 8. The proposed EG-CRNN initially consumes energy, but gradually with the increasing number of nodes the consumed energy becomes less. Therefore, it can be considered that EG-CRNN has the capability to achieve good system performance than DRNN. The system residual energy is one of the most important parameters, in terms of improvement the QoS. This implementation is illustrated in regard to QoS in terms of Traffic Prediction (TP), Packet Loss Ratio (PLR), Buffer Usage Ratio (BUR),and Network Lifetime (NLT).

A. TRAFFIC PREDICTION (TP)
First of all, we implement the proposed network without a controller. The traffic flow getting to the BS is shown in Fig. 9. It is clear from this figure that the traffic flow is exceeding the capacity of the FIFO buffer. Fig. 10 illustrates the comparison of the real tf and the estimatedf (tf ) delivered by the network after applying the AIS. It can be seen that the EG-CRNN can estimate the traffic of the network as the actual traffic is very close to the estimated one. At the beginning of the simulation, the actual traffic rises to a high  level until the controller starts to work to predict the traffic and to reduce the congestion, and after a period of time has passed, the actual traffic follows the estimated amount. Fig.11 shows the same comparison with the exception that DRNN is used as the congestion controller. It is clear from Fig. 11 that DRNN is not a good estimator of the traffic, because the arrived packets are more than the capacity of the buffer. The performance of EG-CRNN is better than that of DRNN, a thing which is obvious when the network maintains its traffic within the buffer size of CH. There are four CHs in this emulation. Once they are active, the network using the proposed structure can work in high traffic flow and can control the traffic so as to eliminate congestion at the queue. EG-CRNN has a good ability to predict the traffic as compared with DRNN. This is because the proposed training algorithm has enhanced the performance of EG-CRNN. Also, the addition of the EG element to the proposed network strengthens its long and short term memory. In sum,  the proposed system enhances the ability of the network to learn during offline training, which depends on previous traffic flow in order to predict the current traffic flow.

B. PACKET LOSS RATIO (PLR)
The packet loss ratio can be defined as the ratio of the number of packets loss by the network to the total number of packets generated by the active sensor nodes. Fig. 12 explains the packet loss ratio in the I-IoT network, when the EG-CRNN is applied. A comparison between the EG-CRNN and DRNN is presented in Fig.12. We can see from Fig.12 that the PLR of the proposed system is better than that of DRNN, because the prediction controller and CH assignment have the ability to reduce the sending rate of the effective clusters during the transmission mechanism. The loss ratio of the packets in the proposed system is within 7% of the total packets sent while with DRNN the ratio is 8.5%. This means that the proposed  system can estimate the traffic as well as the capacity of the buffer much better than DRNN. Fig.13 depicts the buffer usage ratio of the network using EG-CRNN as compared with DRNN. This ratio is defined as the number of packets generated by the CH to the capacity of the buffer of the data gathering node. It is clear from Fig.13 that the controlled network guarantees good buffer usage ratio. It is also noted that the buffer of the CH in EG-CRNN does not exceed its capacity. In fact, the values of BUR are less than 95% of the capacity of the buffer. This indicates that the controller can achieve good usage of the buffer in terms of high throughput and less congestion. The recurrence of the proposed system gives it the power to predict the traffic more efficiently as compared with DRNN.

D. NETWORK LIFETIME (NLT)
A big concern with IoT networks is how to maximize the network lifetime while controlling the coverage requirement [36]. Fig. 14 shows the comparison of the lifetime of the network when the EG-CRNN, DRNN and CNN are implemented. It is obvious that the proposed system extends the life of the network more than DRNN and CNN by 12.7% and 22.05%, respectively. The measurement of the lifetime of the network adopted here is the time needed ti take out the energy of all the IoT sensors in the IoT network. The mobile data gathering node that is used in the network has the affordance of extending the network lifetime, i.e. the sensors can retain their energy for a longer period of time. Table 3 shows the numerical results of the NLT and the number of epochs of RMSE during the offline training phase. It is also clear from Table 3 that the proposed structure has better results from other structures, viz. DRNN and CNN.

VII. CONCLUSION
This paper proposed an Intelligent-IoT architecture to be utilized in health care applications. Also, this paper proposed a deep learning AIS as a controller, which has the ability to estimate the flow of the packets of each cluster in the network. This controller, based on EG-CRNN, has an element wise attention gate and self-feedback to predict the traffic. The proposed system is also able to select the CH and its members. The emulation results show that adding the EG to CRNN substantially boosts network performance and hence, the QoS is improved in the I-IoT platform. The AIS provides the ability to select the CH and its members efficiently, which is illustrated in the results of QoS. In addition, the mobile data gathering node proposed in this paper is effective in terms of enhancing the QoS. The RMS error is decreased by 37.14% with EG-CRNN as compared with DRNN and hence, the QoS is enhanced, e.g., the NLT is improved by 12.7% over DRNN.
Controlling the network by EG-CRNN is much more effective than with by DRNN owing to the element wise attention gate that empowers the training algorithm to reach the error goal in a timely manner.