Cloudlet Selection in Cache-Enabled Fog Networks for Latency Sensitive IoT Applications

Over the coming years, the foresighted enormous increase in smart devices supporting Internet-of-Things (IoT) applications demand novelty in network design. A promising solution to the ever-increasing low-latency requirement of IoT applications is the development of fog network architecture. However, the presence of an enormous number of smart devices in fog networks affects the performance of the network. To harvest the benefits of fog networking necessitates finding optimal cloudlet selection strategies. This article formulates a mixed-integer non-linear programming (MINLP) problem that has the objective of latency minimization. An exhaustive search on our cache-enabled (CE) fog architecture cannot be applied because of the problem’s combinatorial and NP-hard nature. Similarly, the genetic algorithm (GA) cannot be used to find the solution because of the calculation of the number of generations. The increase in the number of IoT and fog nodes increases the solution search space, hence an Outer Approximation Algorithm (OAA) is proposed to arrive at the solution. Low complexity, convergence, and effectiveness of the proposed algorithm ensures the $\epsilon $ -optimal solution = 10−3, obtained through standard problem solvers.


I. INTRODUCTION
Exponential industrial development in all fields expected to result in an enormous number of smart devices deployment. Prediction is that more than 50 billion smart devices will be deployed which results in the production of 13 times more data than non-smart devices, by 2020 [1]. Deployment of IoT nodes supporting latency-sensitive applications is challenging because of their stringent ultra-reliable, low-latency communication (URLLC) and different quality-of-service (QoS) requirements. This scalability and low-latency requirement of IoT nodes results in the inefficiency of cloud computing. Computation/processing of many different applications at the same time on the cloud server results in added delays. Propagation delays are also added because of cloud placement at the further place. Because of these latency-sensitive applications running on the user's end, an extension of cloud computing named as fog computing has emerged. It will act as a key enabling technology to support the The associate editor coordinating the review of this manuscript and approving it for publication was Alessandro Pozzebon. future 5th Generation (5G) IoT applications [2], [3]. For the fourth industrial revolution, fog computing will be an enabling technology because it supports mobility, locationawareness, heterogeneous IoT smart devices, realtime and secure communication [3], [4], [5]. Fog computing is a decentralized solution for latency-sensitive applications, which involves the computation and storage resources of data near the edge of the network, near to the IoT nodes. Cloudlet nodes are an extension of a cloud server placed close to the IoT nodes. In comparison, cloudlet nodes have limited data storage capacity and computation/processing capability than the cloud server.
To access, process, and forward information between cloudlet and cloud server, the backhaul link's burden increases. This burden results in propagation and processing delays. Similarly, throughput increases due to sharing a single spectrum and heavy traffic burden between cloudlet node and IoT devices. This results in the fronthaul link's burden, resulting in a transmission delay. Because of the burden limitations of these links, the concept of caching at cloudlet nodes has been proposed. The presence of cache on cloudlet nodes in fog networks will result in the minimization of the overall network's latency [6]. As the requested data available in close vicinity of users at the cloudlet node, there will be no computation and propagation delays. Cache-enabled (CE) cloudlet nodes in fog network will support latency-sensitive IoT applications.
Fog computing as an enabler for industry 4.0 applications has some serious implementation challenges of cacheplacement, resource management, network deployment, network modeling, energy consumption, and user association. For minimizing the overall network latency, efficient cloudlet-selection and user-association techniques for CE-fog networks are required. Recently researchers have done some work on resource management aiming different objective functions of maximizing energy efficiency, minimizing latency, and maximizing network utility in fog networks. Section II provides a detailed literature survey on existing relevant work on the cloudlet selection, mentioning their limitations in Table 1. System model and the problem formulation is explained in the Section III. Section IV discusses the proposed algorithm followed by algorithm's convergence proof and its complexity analysis. Numerical results that are observed under the proposed algorithm are presented in Section V. Finally, we conclude this work in the last Section VI.

II. LITERATURE REVIEW
Globally, many researchers and industrial assemblies have noticed the need for new research techniques for real-time communication supporting IoT applications. In the upcoming future, new techniques for the deployment of networks, management of resources, and energy will be under consideration to minimize the latency for IoT applications. The fog computing paradigm causes the upcoming shift in network architecture that supports reliable, economical, and real-time communication for future IoT applications. Cacheable Small Cell Base Stations (SCBs) are widely deployed in small cell networks (SCNs) for managing traffic load across the network. Network Utility Aware (NUA) load balancing scheme was proposed by authors in [7]. Offloading, buffering, and resource allocation are three parallel algorithms proposed by authors to propose an optimal resource allocation on the basis of throughput, balancing, and delay [8]. A graph-based caching approach with less complexity was proposed in comparison with the brute-force approach in [9]. The proposed approach gives remarkable improvements in terms of traffic offloading parameters. Authors have formulated a 0-1 integer programming problem for fog radio access network (F-RAN) under constraints of maximum distance and maximum traffic load threshold.
A new bio-inspired optimization algorithm named as Bees Life Algorithm (BLA) was proposed by authors in [10]. Their objective function is to distribute tasks among fog nodes optimally. They proposed better results after studying an optimal trade-off between execution time and memory. In [11], authors have proposed an optimal resource allocation scheme among fog nodes (FNs) and data service operators (DSOs). First Stackelberg game was proposed to analyze the pricing problem and then a many-to-many matching algorithm is used for FN-DSS (data service subscribers) pricing problem. For the joint optimization problem of cloudlet selection and bandwidth allocation, an iterative algorithm was proposed by authors in [12] for a triple-stage Stackelberg game. To achieve maximum utility by minimizing latency during association authors have proposed the Boltzmann-Gibbs learning algorithm [13]. User clustering was done based on the user's request and using the proposed algorithm was used to choose a cache-enabled cloudlet. Similarity-Aware popularity-based Caching (SAPoC) is proposed in [14] to improve the performance of the network in terms of the cache hit ratio and energy consumption. The arrival and departure of mobile devices are considered for the wireless edge computing network. If the requested file is already stored at the edge of the network, the cache hit ratio is triggered, otherwise, the request is sent to the cloud for processing. The freshness of stored content, similarity, and frequency of request is determined by content popularity prediction. The performance of the proposed algorithm is compared with other caching algorithms, in terms of energy consumption and cache hit ratio. In [15], authors have proposed a solution for new emerging Mobile Edge Computing (MEC) systems using the GNU Linear Program kit (GPLK) for the formulated problem. User association and admission control were studied under delay and cloudlet's storage capacity. A tradeoff between queue delay and power consumption was investigated for the proposed F-RAN scenario. The stochastic-based mixed-integer, joint optimization on mode selection and resource allocation problem is formulated. Authors have studied two reinforcement learning-based (RL) algorithms and solved the problem using the Lyapunov optimization [16].
For resource management, in [17] authors have proposed an energy-aware algorithm. and an evolutionary algorithm for less energy consumption respectively. A tradeoff between power consumption and delay was studied in [18]. The authors consider the power and delay constraints for F-RAN architecture. They proposed a delay-aware energy-efficient computation offloading scheme for minimizing the consumption of grid energy [19]. A power control algorithm and Greedy algorithm (GA) was studied with the aim of energy-efficient resource allocation in [20] and [21] respectively. To identify a suitable node for caching in a distributed way, a content-placement algorithm named as Content-Based Centrality (CBC) was proposed in [22] for information-centric fog computing. Authors have formulated a non-convex optimization problem aiming maximization of content at fog nodes subjected to buffer size threshold.
To maximize the overall delivery rate for cache-enabled F-RANs architecture, a convex approximation method was studied by authors in [23]. The joint optimization problem was formulated under fronthaul capacity, file size, and transmit power for user association to fog node. In [24], VOLUME 9, 2021 two different (maximum data rate and minimum latency) optimization problems were studied. A distributed algorithm is proposed for solving the first joint problem of caching placement and association strategy. Optimal power allocation is derived in a closed-form expression for the second joint problem of mode selection and subchannel assignment. Resource management under storage capacity and bounded delay constraints to minimize latency were studied in [25] and [26]. The optimal workload allocation problem under optimal tradeoff between latency and power was studied in [27]. Authors have proposed two distributed algorithms supporting novel fog node cooperation strategy. A self-organized many-to-one matching game approach was used in [28], [29], and [30] to minimize the latency during cloudlet selection. Computational caching algorithm and Regret-Based algorithm were proposed by authors in [31] and [32] respectively, to a latency minimization problem under a storage size constraint. For load balancing, user association in [33], [34], [35], authors have studied cloud-fog integrated Industrial Internet of Things (CF-IIoT) network to get ultra-low latency. For this objective, they have proposed a real-coded genetic algorithm for constrained optimization problem (RCGA-CO) algorithm. They have formulated a min-max optimization problem for resource management. Table. I summarize the mentioned related work mentioning respective contributions.

A. CONTRIBUTIONS
The key motivation of our work is that none of the previous work considered joint resource selection, admission control, and power allocation scheme for a cache-enabled (CE)-fog network that minimizes the overall network latency. This work is the first attempt that provides a solution for the future upcoming industrial revolutionized era which will be composed of real-time low latency IoT applications. To the best of our knowledge and comparison of some previous work mentioned in Table 1, our contributions in this article are: 1) We propose a mathematical framework for latency minimization in the CE-fog network. 2) A joint optimization problem is presented, which is of mixed-integer non-linear programming type. The formulated problem jointly takes into account cloudlet selection, admission control, and power allocation parameters. 3) Due to the combinatorial and NP-hard nature of the problem, problem complexity will increase as the number of integer variables increases. In this article, with finite convergence, an -optimal solution is achieved using the outer approximation algorithm (OAA). 4) We also compared the performance of OAA algorithm with the genetic algorithm (GA) and the exhaustive search algorithm (ESA). The results show OAA gives better results than both. 5) Extensive simulations have been done to find -optimal solution using the low-complex OAA, which means that an optimal solution is just only = 10 −3 away.

III. SYSTEM MODEL AND PROBLEM FORMULATION
A proposed fog network is shown in Fig. 1 in which there are g number of cloudlets or fog nodes such that g ∈ G = {1, 2, . . . , G} and a single cloud is deployed. There is n number of IoT nodes in the network such that n ∈ N = {1, 2, . . . , N }, connected to cloudlets via a wireless link. To share load and information, cloudlet nodes have dedicated links among each other. To mitigate the interference between cloudlet-to-cloudlet link and IoT node-to-cloudlet node, the frequency bands are chosen that are different or non-overlapping. Dedicated backhaul links with high-speed capability are used for the connection of  cloudlets with the cloud server. The same frequency band is used for communication between cloudlets and IoT nodes, hence, there is interference experienced by IoT nodes from other close vicinity cloudlets. Each cloudlet g with a limited capacity of S g bits is equipped with a set of data files given as c ∈ C = {1, 2, . . . , C}. IoT node n requests a file content c with the size of d c bits, this request arrival rate has a Poisson distribution. This distribution has a mean of λ n,c where a higher mean arrival rate of a file shows that it is more popular in the system. If requested file c is available at the cloudlet's cache, it is directly delivered to the node, otherwise, the request will be handled by the cloud itself. This cloud request will increase in latency and overall system cost. At a particular time, a cloudlet can serve multiple IoT nodes but a node can only be connected to only one cloudlet g. Let an IoT node n ∈ N be present in the coverage of multiple cloudlets and intend to download some files. Let x n g = {0, 1} ∀n ∈ N , g ∈ G be the binary indicator representing association of node with a cloudlet. It has value 1 when there is a connection between the node and a cloudlet, otherwise its value will be 0. Let y n g = {0, 1} ∀n ∈ N , g ∈ G be the binary indicator, which decides whether a node is admissible for connection or not, such that, x n g ≤ y n g . IoT node admission will be done based on QoS requirements, given by a cloudlet to a user node. Variable y n g , represents admission control on cloudlet nodes, while x n g is a node association variable which ensures that a user node will be transmitting data with only one selected cloudlet. If an IoT node is admissible on a particular cloudlet node it has value 1, otherwise, if rate and latency constraints are not satisfied its value will be 0. Let z n g = {0, 1} ∀n ∈ N , g ∈ G be the binary indicator that shows the availability of requested file c in the cache memory of a cloudlet g. The proposed system model is shown in Fig. 1. We define total communication latency experienced by node n to get the requested content c from cloudlet g as l n g that can be stated as: where l T g,n and l B g,n are wireless transmission delay and backhaul delay, respectively. These are the delays experienced by an IoT node n after requesting files from the associated cloudlet node g. During transmission of a file c with size d c from cloudlet g to IoT node n, the transmission delay of the wireless link can be calculated as: In practice, traffic load and average link distance are related to backhaul delay, which is greater than the value of the transmission delay. l B g,n is a combination of fog processing delay l P g,n and propagation delay l D g,n . Here, for simplicity, we assume that processing delay for all fog nodes has a fixed value. If the IoT node received the requested files from the CE-associated cloudlet node, there will be no backhaul delay. However, if the requested file is not available in the cache memory of the cloudlet, the IoT node will experience both the delays as mentioned in Eq. (1). First, there will be a transmission delay and then there will be a backhaul delay, as the cloudlet has to fetch the requested files from the cloud via backhaul links. These delays will cause an overall increase in latency experienced by an IoT node. In Eq. (1), the backhaul delay l B g,n experienced in fetching files from the cloud can be calculated as [25]: The propagation delay l D g,n is dependent on the D b , which is the distance between the fog node and cloud. We have assumed that fog nodes are placed at different distances from cloud causes different propagation delays, which is given as where r n g is the downlink (DL) transmission rate that is given by Shannon's capacity formula as: here b n g is the assigned bandwidth by cloudlet to the node and SINR n DL,g is the signal-to-interference plus noise ratio experienced by IoT node given as SINR n DL,g = x n g p n g h n g g =g∈G p n g h n g +N o ; p n g and h n g are the assigned power and channel gain, respectively. While power and channel gain received by other interfering IoT nodes are given as p n g and h n g , respectively. Channel gain is defined as , where other parameters are antenna gain (G o ), log-normal shadowing (ζ 10 ζ /2 ), the distance between cloudlet and IoT node (d g ), the antenna far-field reference distance (d o ), path loss exponent (α), Rayleigh random variable (h n g ) and ζ is given as the zero-mean Gaussian random variable with standard deviation σ [36]. A summary of the symbol notation used in this article is given in Table 2.
For downlink transmission under cache-storage capacity, power allocation and QoS constraints, the formulated problem of joint cloudlet selection and latency minimization for CE-fog network, with objective function J , can be mathematically stated as: ∀n ∈ N where x n g ≤ y n g ; ∀n ∈ N , g ∈ G C3 : n∈N p n g ≤ P g ; ∀g ∈ G C4 : p n g ≤ y n g P g ; ∀n ∈ N , g ∈ G C5 : n∈N z n g d c ≤ S g ; ∀g ∈ G C6 : l n g ≤ y n g L; ∀n ∈ N C7 : r n g ≥ y n g R; ∀n ∈ N C8 : p n g ≥ 0; ∀n ∈ N C9 : x n g , z n g ∈ {0, 1}; ∀n ∈ N , g ∈ G At a time, user association constraint as C1 ensures that any node n can be connected to only one cloudlet. QoS constraint as C2 ensuring the user admission to a cloudlet that whether the user gets services from the associated cloudlet or not, based on minimum data rate and latency. C3 is the maximum power constraint that power assigned to all connected nodes must not exceed the maximum transmit power of the cloudlet (P g ). If the node is not connected to any cloudlet its power will not be considered, this is given in C4. C5 is the file caching constraint in terms of the limited storage capacity of a cloudlet (S g ). The total data sent by all users associated with a particular cloudlet must not increase the total storage capacity. Latency constraint as C6 ensures that latency must not exceed the threshold upper bound latency value (L). Constraint C7 is the minimum data rate threshold constraint, ensuring that the associated cloudlet provides a minimum data rate (R) to its admitted user node. Constraint C8 ensures that the power of a particular user that is connected to a single cloudlet must be greater than 0. C9 is the constraint that limits the value of node association and cache placement indicator to binary values of 0 and 1.
The formulated problem falls in a class of mixed-integer non-linear programming (MINLP) problem that is NP-hard. It is impossible to find an optimal solution in polynomial time because of the NP-hard nature. As the number of IoT and cloudlet nodes increases, the search space to find a solution increases exponentially. For a global optimal solution, an exhaustive search algorithm (ESA) can be used, but the search on binary variables results in high complexity. It gives search space having an order of 2 |n| , which means there are 2 |n| optimization problems that need to be solved. Because of the exponential increase in search space, ESA cannot be applied to find a solution. Therefore, we propose the outer approximation algorithm (OAA), which requires relatively low computations to find a near-optimal solution [37]. In the literature, there are some other algorithms to solve the MINLP problem used in literature namely the branch and reduce (BR) algorithm [38], and the method by Lawler and Bell [39].

IV. PROPOSED TECHNIQUE
The Eq. (6) is very complex as it is a combination of binary, continuous, and integer variables. This combination makes a big challenge to solve this problem. With the increase in the number of IoT and cloudlet nodes, many variables need handling which makes it NP-hard. To solve the non-linearity and integrality of MINLP, OAA is applied, which works with the convergence of upper and lower bounds. MINLP is solved after decomposition in the primal problem (NLP) and master problem (MILP). An upper bound is achieved after fixing the binary variables (x n g , y n g , z n g ) which is further used to find a lower bound. The detailed implementation of OAA for the proposed latency minimization problem is given in the next section.

A. OAA DESCRIPTION
In Eq. (6), let the constraints from C1 to C9 that are subject to objective function J , be denoted by a set as F c1−c9 , P = {p n g } as a set of continuous variables and = ϕ ∪ P as set of discrete variables. It is observed that the following four propositions are satisfied by the formulated problem: 1) P is a compact set of variables having properties of non-emptiness and convex. 2) J and F c1−c9 are continually differentiable for making P convex.
3) The NL problem can be obtained after fixing the value of . VOLUME 9, 2021 4) After fixing the values of all discrete variables, the solution of continuous primal problem satisfies all the constraints. OAA will converge in an infinite number of iterations with convergence capability [37]. OAA follows a non-decreasing lower bound and a non-increasing upper bound. After solving the primal-subproblem, the upper-bound sequence is calculated, whereas the lower-bound sequence is calculated after solving the master problem. According to proposition 4, the primal problem is formed using fixed values of . The primal problem with n integer values, at the n th iteration of the algorithm, can be written as: The master problem is formed by the solution (P n ) of the above primal problem. After applying OAA on the Eq. (6), the solution of the primal problem sets the upper bound while the solution of the master problem sets the lower bound. This results in the linearity of both functions [40], [41]. Every next iteration uses integer variables (n+1) i.e solution of master problem. After the result of every iteration, the bounds get close to each other. A point comes where the difference between bounds remains less than , the algorithm stops. We can rewrite the problem as: subject to : F c1−c9 ( n , P) ≤ 0 The master problem in Eq. (9) is the projection of our formulated optimization problem in Eq. (6) on space discrete variables. There is a constraint qualification for every primal problem solution n , which implies that the projected problem will have a similar solution written as: Equivalent minimization problem with new variable W can be written as: min ,P,W W subject to : W ≥ −J ( n , P n ) − ∇J ( n , P n )( P−P n − n ) F c1−c9 ( n , P n )−∇F c1−c9 ( n , P n )( P−P n − n ) ≤ 0 Lower bounds are calculated using the solution of the master problem Eq. (12). This master problem is equal to the formulated problem in Eq. (6), only if all the mentioned propositions are satisfied. The Eq. (12) is of MILP type and can be solved using an iterative approach. The pseudo-code of the proposed OAA is given in Algorithm 1.

B. ALGORITHM CONVERGENCE AND OPTIMALITY
Proof of linear convergence rate of OAA is given in mixed-integer programming literature [41]. The branch and bound architecture make OAA optimal in = 10 −3 . In this procedure, discrete values of are fixed which means that any combination of nodes and cloudlets will never be used twice. If all the four propositions are satisfied and there are a limited number of discrete variables , then Algorithm 1 terminates with the optimal solution in finite steps [37]. The solution is guaranteed using -optimal algorithms within the of the optimal solution for any > 0. The guaranteed accurate value of the solution is given by lower values of . For a specific choice of discrete variables n , the optimality of P in master problem Eq. (12) might be: The algorithm eliminates such values of n for which there is no feasible solution that exists for the master problem. This results in finite algorithm convergence. For any fixed values of , the algorithm optimality follows from the convexity of the objective and constraint function. A detailed convergence proof of the OAA algorithm is given in [42]. Using ESA for Eq. (6), a globally optimal solution can be calculated, but there is an exponential computational load increase as it has to enumerate all nodes and fog selection options. Denoting C ESA as complexity and k as the number of nodes (in our case: fog+IoT) in the network, the computational complexity of the ESA will be given as: Similarly, the stochastic nature of GA makes it more complex. Its complexity depends on the operators and their implementation. The C GA , complexity of GA will be given as: here σ is the number of generations [43]. Space complexity is twice the population size (in our case fog+IoT nodes). But in an infinite number of iterations using OAA, anoptimal solution can be found [37]. In a generalized way, the complexity C OAA , for OAA will be given by: where ρ is the number of constraints and error tolerance of -optimal solution from the global optimal solution is given by γ . One more advantage of OAA over ESA is that it ensures to provide an -optimal solution. In this paper, we compare OAA results with the results obtained 93230 VOLUME 9, 2021 from GA. After benchmarking GA results, it can be seen that the performance of OAA is better than GA. GA could not perform well, as it cannot guarantee any optimal or -optimal solution. Also, there is no convergence proof for GA. The computational complexity trend of ESA and OAA is presented in Fig. 2. The calculation of the number of iterations for GA is not possible because of the statistic nature which depends on the number of generations (Eq. (14)).

Algorithm 1
Outer Approximation Algorithm 1: n ← 1 2: Initialize n 3: ← 10 −3 4:Convergence ← FALSE 5:While Convergence == FALSE do 6: P n ← arg min, −J ( n , P); subject to, F c1−c9 ( n , P) ≤ 0 7: Upper_Bound ← J ( n , P * ) In this subsection, we will be calculating OAA complexity more specifically for our formulated problem. The total number of flops F gives the complexity of an algorithm. In [44], a real floating-point operation is used to represent a flop. Every operation has its corresponding number of flops such as one flop for the operations of addition, multiplication, or division; one flop for additive and removal operator; one flop for a logical operator (e.g. comparison etc.) and assignment operator; two flops for complex addition, four flops for complex multiplication, and 2mno flops for matrix multiplication having mxn and nxo dimension. To find the complexity of the proposed algorithm for our scenario, we have to count the number of flops. In our proposed algorithm, the first five statements take one flop each. Statement 6 takes two flops as 2NG, statement 7 and 8 take 4NGβ flops each, statement 9 takes 2NGβ, statement 10 takes two flops, statement 11 takes one flop, and statement 13 takes two flops. The total flop count F OAA for our system having N number of IoT nodes, G number of fog/cloudlet nodes and β is a constraint count for each node, mathematically given as:

V. SIMULATION AND RESULTS
To solve Eq. (6) and calculate the overall latency of the network, experimental validation is done for the proposed system model. The experimental results portray the performance of the OAA approach. The results also give some insight into the convergence of the proposed algorithm.
To implement the outer approximation linearization technique, basic open-source nonlinear mixed integer programming (BONMIN) [45] solver is used.

A. SIMULATION SETUP
The simulation assumptions of parameters are summarized in Table 3. For all the simulation maximum coverage distance for each fog is set to 50m. The maximum transmitted power for a fog/cloudlet node P g is set as 41dBm. The minimum data rate and latency requirement for any node is set to 200kbps and 1ms, respectively. Reference distance as per antenna far-field d o is set to 5m and d g is always greater than d o . Path loss exponent α is set to 2 and zero mean gaussian variable for shadowing ζ is set to 10dB. The minimum IoT nodes allowed are 2 in the network, whereas the maximum IoT nodes allowed are 18 with an increment of 2. IoT nodes are supposed to be uniformly distributed in the network. Minimum fog nodes are 3, whereas maximum fog nodes allowed are 5, with an increment of 1. These fog nodes are at different distances from the cloud, resulting in different propagation delays. The processing delay at the fog node is set as 0.1ms. The storage capacity for cached data is set as 40MB.

B. DISCUSSION ON RESULTS
In this work effect of power, cache size, and QoS requirements were evaluated using system-level simulations. In the end, we compare the performance of OAA with the ESA  [29], [32], [46]. and standard continuous GA [47]. OAA algorithm is used to calculate network latency under constraints. Fig. 3 shows the association when there are five patches of fog nodes available in the system. The number of associated IoT nodes with fog nodes versus total nodes in the fog-IoT network is observed. Node gets associated with a particular fog node based on the best channel which gives the minimum latency. If an IoT node doesn't receive the minimum QoS requirements based on constraint C2, it is not admitted by any fog node for transmission, and hence such nodes do not contribute in the calculation of overall network latency. With an increase in IoT nodes in the network, user association is maximized while keeping the QoS constraints (C4, C6, and C7) into consideration. The admission of nodes at fog nodes is a random pattern, depending on QoS (power, latency, and rate) requirements. The user association is done with an aim of latency minimization objective (Eq. (6)). Fig. 4 depicts network latency relation with the number of IoT nodes, in presence of different numbers {3-5} of cloudlet nodes in the network. The total system latency is calculated using Eq. (4) and Eq. (6) considering all the constraints. For a fixed number of fog nodes present in the network, the increasing number of IoT nodes increases total system latency, which is obvious as more time is required to serve all nodes. Observing the effect of the number of fog nodes on latency, at the start, the number of fog nodes in a system has no effect on latency as the number of IoT nodes in the network is less. But as the number of IoT nodes increases, the presence of more fog nodes depicts effects on the total latency. For three fog nodes, all nodes have to be served with only these available three fog nodes, resulting in more latency. However, if there are five fog nodes in the system, a positive effect can be seen. The total network latency decreases as there are more resources available for downloading the requested files. For three fog nodes, 2 IoT nodes will incur 0.01ms latency, while for 18 IoT nodes will incur 0.2ms (around 95% of the increase in latency). This proofs the fact of increasing latency with an increase in the number of nodes in the system. The calculated latency values show that the proposed network model can be used to support real-life applications in the future, for example, smart city, healthcare, and industrial floor. The typical QoS (delay) requirements of internet traffic are: RT data (1ms-1s), image(1s), audio and video (0.25s). Similarly based on the application scenario the forecast QoS delay requirements are: industry 4.0 (≤5ms), internet of energy (≤200ms), big data streaming (≤100ms), smart city(≤10ms), factory automation (0.25-10ms), healthcare (1-10ms), robotics, and virtual reality (1ms). Similar more use cases with delay and data rate requirements can be seen in [48], [49]. Table 2 in [3] also summarized the use of fog computing to support the application case studies. Fig. 5 depicts the relation between the number of IoT nodes and mean user throughput. For a specific node, the throughput is almost the same satisfying the minimum rate requirement. As the number of fog nodes increases, it is observed that there is an increase in instantaneous power   which gives more throughput to the user. The user will get more options for receiving the required files, hence the mean user throughput increases with more resources available. This relation i.e., Eq. (5) depicts the direct relation between capacity and power. Figure shows the relation between the number of IoT nodes in the system and the mean user throughput of an IoT node. The graph shows that the mean user throughput is almost the same satisfying constraint C7 of the problem (Eq. 6). The increasing total system throughput can be evident from Fig. 6 in relation to increasing IoT nodes in the system. More IoT nodes will generate more data as compared to less number of nodes. Similarly, the system throughput is maximum for five patches of fog nodes as there will be more resources available for allocation. IoT nodes have more options to fetch their requested files. The sum of the data rate received by all associated IoT nodes in the system gives the total system throughput. The data rate of a single user is calculated using Eq. (5). Precisely, we could say that by increasing the number of IoT or fog nodes into the network, the overall system throughput increases. But there will be a point comes, when any further increase in the number of IoT nodes will not affect the total system throughput, at this point the system's throughput reaches its maximum capacity limit. After this point, the system throughput will be constant. Fig. 7 shows the effects of cache size on total latency. Total system latency is calculated using the objective function of the formulated problem (Eq. (6)), under different sizes of the storage capacity of fog nodes given in constraint C5. The figure depicts that if the number of IoT nodes increases in the network there will be an increase in system latency. As more number of IoT nodes need more time to fulfill their requirements which causes more delay, hence increasing the overall system latency. To observe the effect of cache size in the figure, it is evident that if there is no cache available at the fog node, the system experiences maximum latency. A node has to experience all the delays namely; transmission, processing, and propagation. A fog node will have to send the request to the cloud server using backhaul links. The sum of all delays results in a maximum latency calculation that does not satisfy the maximum delay threshold for an IoT node. The presence of cache at the fog node has clear positive effects on latency, resulting in the efficient performance of VOLUME 9, 2021 the network. For 20MB capacity, latency is minimum as compared to no − cache, as some demanded files from nodes are available at fog. As we increase the size of the cache at the fog, overall system latency decreases. Table 4 shows the percentage reduction of latency for various cache memory sizes in relation to no-cache. The percentage reduction is more for 60MB size capacity as compared to 20MB. This is because more number of files are present near the vicinity of IoT nodes. At first latency reduction increases, but as the number of IoT nodes increases in the network, the latency reduction decreases. This is due to the fact of being served at the same time, more IoT nodes in the system will take more time.
In Fig. 8, the performance of the network in terms of latency calculation is observed under different algorithms; namely OAA, GA, and ESA. For all algorithms, the parameter assumptions are the same, and latency is calculated using Eq. (6). We can see the performance comparison of the OAA with ESA and GA, for five patches of fog nodes. OAA performs better because of its -optimal nature over GA. The GA can give good results, but there is no guaranteed optimal solution and convergence proof because of the stochastic nature of GA. When there is a fewer number of IoT nodes are present in the network, the performance of all algorithms is almost the same. But with the increase of IoT nodes, the algorithm's behavior starts some different behavior. ESA is the most complex and the time taken algorithm gives more latency than OAA and GA. In comparison with ESA, OAA is less complex and converges in a finite number of iterations. ESA might give the best optimal solution with a trade-off of the algorithm's complexity. Complexity comparisons of all algorithms in a theoretical way are previously discussed in Section IV-B. The choice of the algorithm also has effects on latency calculations, ESA and GA take more time in finding the optimal solution. This behavior can be numerically observed using equations (Eq. (13), Eq. (14), Eq. (15)).

VI. CONCLUSION AND FUTURE DIRECTIONS
In the very dense IoT application scenario, an extraordinary burden is on the cloud every time. In this paper, CE cloudlet nodes are integrated into the system to reduce the overall latency of the network. A joint MINLP optimization problem is formulated which considers resource association, IoT node admission, power allocation, and cache-availability constraints. The IoT-CE cloudlet system was studied for the minimization of latency. A less complex, branch and bound OAA technique is used to find an − optimal solution. Extensive evaluation results show the effectiveness of the proposed approach for our system. A comparison is made in terms of storage capacity at fog nodes and the total number of fog nodes. An increase in intelligently placed data storage at fog node and overall fog nodes in a system deployment results in the minimization of system latency. Designing and optimization of future networks with low latency, more energy-efficient attributes is a vast area of research and many questions need to be answered. For future work, we will find an energy-efficient resource allocation strategy for the proposed system model. We will upgrade the current problem with the reliability factor. For future, mission-critical IoT applications, this ultra-low latency and reliable communication (URLLC) is the major requirement. In this work we have used the standard optimization technique (OAA) to solve the problem, for the future, a heuristic approach can also be used.
RABEEA BASIR received the bachelor's and master's degrees in telecom engineering from the University of Engineering and Technology, Taxila, Pakistan, in 2013 and 2015, respectively. She is currently pursuing the Ph.D. degree with the School of Electrical Engineering and Computer Science, National University of Sciences and Technology (NUST). Her research interests include fog networking, quality of service provisioning, and the IoT networks.