Multi-Level Resource Sharing Framework Using Collaborative Fog Environment for Smart Cities

Fog computing has proved its importance over legacy cloud architectures for computation, storage, and communication where edge devices are used to facilitate the delay-sensitive applications. The inception of fog nodes has brought computing intelligence close to the end-devices. Many fog computing frameworks have been proposed where edge devices are used for computation. In this paper, we proposed a simulation framework for fog devices that can use end devices to handle the peak computation load to provide better Quality of Services (QoS). The regional fog nodes are deployed at network edge locations which are used as an intelligent agent to handle the computation requests by either scheduling them on local servers, cloud data centers, or at the under-utilized end-user devices. The proposed device-to-device resource sharing model relies on Ant Colony Optimization (ACO) and Earliest Deadline First(EDF) Algorithm to provide a better quality of service using device available at multi-layer design. The concept of using IoT devices as fog nodes has improved the performance of legacy fog based systems. The proposed work is benchmarked in terms of system cost, efficiency, energy, and quality of service. Further, the proposed framework is with xFogSim in terms of task efficiency.


I. INTRODUCTION
With greater needs for sensor devices and the internet of things, a huge amount of data is produced which requires extensive computing resources. The concept of 5G has further increased the data generation rate with high-speed data rate between the device-to-device communication is used. Therefore, compute extensive tasks can be offloaded to cloud data centers with an additional cost of multi-hop network latency, end-to-end delay, bandwidth, and congestion. However, to facilitate the delay-sensitive tasks such as medical, power, gas and oil leaks, the concept of fog computing is adopted that helps in performing task computations on near fog devices. Thus, reducing the network load while sending the data to cloud data centers. This inclusion helps in reducing task latency, end-to-end delay, and bandwidth. In a multi-agent fog environment, the concept of fog broker is used which manages the local fog nodes, defines task execution policies, and communicates with other devices [1].
The associate editor coordinating the review of this manuscript and approving it for publication was Zehua Guo .
A generic layered architecture of the fog computing environment is given in Fig. 1. There are three layers in conventional fog networks. The lower level comprises IoT devices that request resources from gateway broker nodes. The second step comprises gateway nodes that connect the end-user devices with fog nodes and also perform resource management. In case, the gateway broker does not have vacant resources it requests nearby brokers for resources. The broker returns the task results to IoT devices. The application that can be facilitated through fog nodes is security systems, healthcare systems, or facial detection, etc. The gateway broker can schedule the task to local resources; however, doing that constantly can result in long queues which can degrade the system performance [2].
In recent years, most of the work has been focused on scheduling algorithms, task distribution, and the energy preservation of handheld devices. The adoption of 5G has changed this concept of using central nodes for compute-intensive tasks execution. In 5G, device-to-device communication is used to achieve high throughput, low power consumption, and wide-area coverage [3]. In the proposed work, the concept of device-to-device communication is used which enhances overall system performance and reduces waiting time as well as network congestion [4].
In this article, we proposed a simulation framework for fog networks that inherit the concept of the device to device communication to offload the task to the nearby end device. These end devices are named as volunteer nodes because they volunteer their resources and communicate directly with task requesting nodes to achieve the device-to-device communication feature of 5G. There are gateway nodes act as a broker node to resolve the incoming computing requests. The dedicated fog devices and volunteer devices are used to perform computation. The framework is dynamic, devices having sufficient energy, and available resources can participate in the resource sharing model where the broker nodes allocate requests to the most capable device, keeping in view the energy and other constraints. When a task is performed, the volunteer computing device sends results directly to the requesting device and acknowledges to the broker device. The other salient features of the framework are: • Efficient energy management module which enables the system to preserve energy through task scheduling over devices having significant energy.
• Safe handover mechanism under mobile environment, especially, When both end user device and volunteer fog nodes are mobile, the localization becomes very complex.
• Configurable mobility module that allow researchers to include their own mobility models through the extensive markup language.
• Inclusion of collaborative geographically distributed fog locations to ensure the availability of resources.
• Addition of backend datacenters for batch processing tasks having less stringent deadlines, and computation intensive.
• Quality of service (QoS) and resource localization based allocation algorithm to ensure that the task gets its required QoS with least possible communication delay to improve overall system performance. The rest of the paper is structured as follows: Section II covers the related work. The system model is presented in Section III. Section IV covers the proposed framework in detail and system evaluation is covered in Section V. Finally, the conclusion and future directions are presented in Section VII.

II. LITERATURE REVIEW
This section covers the existing work on fog computing frameworks and simulators.

A. FOG COMPUTING FRAMEWORKS
YAFS et al. [5] proposed a simulation framework to enable users to simulate fog networks in the context of node placement. The authors allow users to place fog nodes based workstations at different network topologies to perform result comparison.
Belli et al. [24] proposed a fog based mobile crowdsensing platform for the efficient users' recruitment process. They utilized the concept of using mobile nodes to perform specific tasks in a complex fog network. The mobile nodes advertise their battery and CPU power and the fog node takes the offloading decision. However, the fog nodes are not utilized for computation purpose; thus, only act as a task distributor.
Tom et al. [25] proposed a routing protocol labeled as Low Power Lossy Area Network for smart metering. The authors incorporated a middle node i.e. aggregator between meters and fog nodes. Using the intermediate node, the authors claim to improve the packet delivery ratio by 35% and decreased end-to-end delay by 13%. The simulation is performed using Contiki OS [26].
Natesha et al. [27] proposed an IoT application module placement strategy for fog network environment. The authors benchmark the proposed work using iFogSim and claim a significant decrease in delay and power consumption.
Huang et al. [28] proposed a mathematical model to generate automated test cases based on path coverage for a dynamic fog environment.
Avaisi et al. [29] proposed an efficient car parking system that uses fog computing concepts to reduce network latency and delay. The solution is tested using iFogSim and compared with a similar solution in a cloud-based environment. Further, significant improvement has been observed in terms of latency and performance.
Rehman et al. [30] proposed a trust-based task mapping and resource sharing framework for vehicular fog networks. They applied the concept of 5G and utilized the concept of trust matrix to form an off-street cluster of vehicles. A task mapping algorithm is also proposed which enhances the efficiency of resource sharing.
Nashaat et al. [31] proposed an IoT application placement algorithm for a fog environment. The authors classified the algorithm into two phases. The first phase is to prioritize IoT application requests based on application usage, user expectation, and environment run-time context. It used QoS violation as feedback, in the second phase it places the request to fog node based on its computing capability, proximity, and its response time. This model showed improved system performance plus a minute increase in power consumption. Dar et al. [32] proposed and designed a delay-aware accident detection and response system. The authors believe that in developing countries, economy scale vehicles are used which are not digitally equipped. Therefore, an Android phone is used where the application detects accidents through its sensors. This information is sent to the nearest hospital.
Deep et al. [33] proposed a 5G optical fog node which creates cyberspace near IoT devices. The devices can efficiently upload tasks to the fog node and gets the response within the time-bound.
Rafique et al. [34] proposed a resource allocation strategy in fog network. A hybrid modified cat swarm optimization algorithm (MCSO) and a modified particle swarm optimization algorithm (MPSO) is used for task allocation at the fog nodes. They distributed resources based on incoming demands. The main objective is to reduce delay and response time.
Ammad et al. [35] proposed a new multi-level energyefficient framework for fog and IoT based smart networks. Two additional layers, energy-efficient hardware, and policy layers are introduced in the IoT-fog-cloud network to make energy-aware decisions. The energy required to execute a particular task is also estimated. Further, various case studies are discussed to evaluate the performance of the proposed framework which includes the smart airport, smart agriculture, smart hospital, and smart parking.
Coutinho et al. [36] proposed a toolkit for fog computing emulation. They designed the fogbed in a virtualized environment which enables users to understand fog concepts and test fog components using third-party available tools.

B. FOG SIMULATORS
RECAP [6], is a fog/cloud simulation framework which can be used to simulate large scale fog, edge, and cloud network scenarios for applications related to control and decision. The simulator provides support in predicting the impact on Quality of service (QoS), workload, and resources.
iFogSim [14] is a simulation toolkit for fog network environments. It adopted the concepts from CloudSim. The iFogSim has three main components, i.e. physical, management, and logical. Buyya et al. [37] elaborated the concept of iFogSim and its usage in various applications.
piFogBed [38] is the first fog computing testbed which is based on Raspberrypi and can easily be configured to use with a realistic fog network. However, for the evaluation, the authors simulated a medical monitoring system.
MobFogSim- [8] is an iFogSim based simulator that enables the modeling of the node's mobility and service migration. In this simulator, the impact of mobility on service migration is studied using an experiment-based approach. MockFog [39] is a freely available simulator where users can simulate small scale fog networks, and test the performance by setting up the network characteristics. Héctor [40] is a simulator for automatic testing of IoT devices. The simulator generates virtual testbeds with customizable network characteristics.
PureEdgeSim [16] is used for performance evaluation of cloud, edge, and fog networks and simulation of resource management strategies. A Fuzzy Decision Tree-based resource allocation algorithm is also proposed which enables the system to adopt IoT environment changes. Finally, a case study is simulated to evaluate the simulator's performance and valuable results are collected.
SatEdgeSim [41] is PureEdgeSim based simulator for modeling satellite edge computing-based networks. The impact of various task deployment strategies is studied. A comparison is also given between conventional task deployment strategies and proposed strategies. The results show that the proposed model is efficient in terms of delay, energy consumption, and task success rate. xFogSim [23] is a simulator designed in OMNeT++ 1 which can be used to simulate distributed fog networks. Medium-scale networks are simulated and it has a very efficient task distribution algorithm that chooses the best fog node depending on the cost, availability, and response time. The simulator allows researchers to simulate complex network scenarios, and evaluate the network performance.
The proposed system has many additional features as compared to xFogSim. In exFogsim, only the dedicated fog devices are used to compute tasks but in the proposed framework the concept of device-to-device communication is applied which increases the available system resources because here some user nodes will be acting as fog devices to compute tasks. An efficient and collaborative resource sharing algorithm is proposed that chooses the best available option for task computation from volunteer, fog, and cloud. The task acceptance ratio is increased because no task is dropped due to a limited number of fog devices. An efficient energy management module is deployed. Table 1 shows a detailed comparison of the proposed simulation framework with many existing fog simulators.

III. SYSTEM MODEL
The system architecture is represented in Fig. 2. There is G n number of fog locations and each fog location is equipped with W m fog gateway nodes. Whereas, the n fog devices are dedicated computing nodes. There are U n heterogeneous devices which include IoT sensors, mobile nodes, user devices, vehicles, roadside sensors. Except for the sensors, all can work in two modes simultaneously i.e. computation seeker and computation provider. These mobile nodes update the gateway nodes about their available energy, and computation power through heartbeat messages. When a user node U i generates computation request x i and send it to nearby gateway fog node W i available in their region. The gateway nodes forward this request to dedicated fog  node i or forward to a volunteer user node V i . If a task is completed on the dedicated fog node, it sends the result to the requesting node directly and acknowledges the gateway node about this task expectation to ensure desirability and avoid data loss. The desirability means whether a task received its desired quality of service or not. The same happens if the task is computed on the user node. In both scenarios, systems ensure device-to-device communication to reduce extra delay and to reduce the burden at the centralized fog node. All this communication is being performed staying in the LAN network. In some cases, the system can communicate with cloud servers through a wide area network.
In the proposed work M/M/c model is adopted [42] where the system follows a queue with multiple computing nodes. However, the arrival rate is simulated using the Poisson process. The total arrival rate λ T at fog device i is given in eq. 1.
where λ i is the average arrival rate at ith fog node . The system will accept new task request if the total arrival rate λ T is less the total workload capacity of the system ν T . According to Liu et al. [43] the decision ι is made by equ 2.
The task execution rate D x i is determined using [44].
The computational delay for a request x i is calculated using eq. 4: Considering the system is first to come first serve based queue and D x i means task execution delay which is a combination of both queuing delay and the service delay.
The notation D n denotes the network delay. There are three possibilities in our proposed work, i). The task is computed on a dedicated fog device in a nearby fog location ii). the task is computed on local dedicated fog devices and iii). the task is computed on a volunteer user node. If the task is executed on a dedicated fog device within the same fog location then the network delay is calculated using eq. 5 where d u, i is the network delay between user and fog device.
The eq. 6 is used to calculate the network delay when a task is executed on a remote fog node.
Here d u, i means the delay between user and local fog device, and d i ,f is the average delay between local fog device and the remote fog device. In eq. 7, the network delay is given when a task is computed on a local volunteer node [45].
This is the best-case scenario where L u i ,w is the latency between ith user and gateway, L u j ,w is the latency between jth user and gateway, and L u i ,u j is the latency between ith and jth user for the jth user result transmission. The distance between a user node and volunteer node is determined by standard distance equation 8 where p(x,y) and v(x,y) are the coordinates of user u and volunteer node v respectively in a two-dimensional plane.
The energy of the system is divided into three categories. i). energy consumed by the access network devices (α) e.g. switches/routers when a request x i of size γ is transmitted in a fog network E α,x i ii). energy consumed by the fog device, volunteer user device, or cloud servers for request processing (ψ) i.e. E ψ,x i . iii). sometimes additional data is required that consume network energy for this request i.e. β * E ρ,x i where β is the ratio N sent /N received of the number of the request sent/received to the cloud. According to Deng et al. [45], the total energy consumed by fog is given in eq. 9.
Here E α,x i is the energy consumed by access network elements to upload request x i to fog device, E ψ,x i is the energy consumption during process and storage operation at fog device, and E α2,x i is the energy consumed by network elements when the result is returned to requesting user. When task is computed at volunteer user device, the energy consumption is computed in eq. 10 The system has a feature of leasing resources from nearby fog locations, so the energy consumption when a request is computed in a remote location is given in eq. 11 where E R α,x i is the energy consumed by network nodes to transmit data to a remote location, E R ψ,x i and E R α2,x i are the energies consumed during processing request and returning result to requesting user respectively. In some rare cases, the request is processed in cloud servers. Hence the energy consumed in that case is given in eq. 12 The access energy consumed by the network elements e.g. switches/routers is calculated in equation 13 The energy consumption for execution of request x i at ith volunteer node is given in eq.14 and the energy consumption during task execution at a fog device i is calculated using eq. 15 In next section, system components are discussed in details.

IV. PROPOSED FRAMEWORK
The proposed simulation framework consists of mobile and static nodes. Let's assume, there are n fog locations. Each fog node has a broker node that can resolve the incoming computing requests. Moreover, there are few mobile devices which act as a fog node to share their resources. Whereas, the requests are generated from user devices that only send requests which are resolved through fog broker. The user device sends job tasks to the fog broker with the strict QoS constraints. The fog broker schedules the incoming computation jobs to a local fog node or volunteer device. A detailed flow of tasks and their control is shown in Algorithm 3.

A. DESIGN COMPONENTS AND UTILITIES
The core components of the proposed framework include user devices, volunteer user devices, fog gateway, centralized fog nodes, and backend cloud data center.
User devices -are the nodes which include IoT sensors, mobile devices, and other handheld devices. These devices communicate with gateways devices to resolve computation requests.
Volunteer user devices -volunteer devices are the mobile fog nodes which can share their resources on request. The volunteer devices broadcast their resource information along with available energies to nearby fog gateway nodes. This received information is used to select the most suitable device for the request. The volunteer node executes the assigned task and returns the result directly to the requesting user nodes. Thus, for result delivery no intermediate node is considered, this is to reduce the delay and network congestion. The task execution at volunteer node is illustrated in algorithm 2.
Fog gateway -is responsible for receiving resources requests, heartbeats messages from the user, and volunteer devices. The gateway selects the suitable device for the incoming request and forwards the task to the selected device accordingly. The potential computation devices include dedicated fog nodes available at fog gateways, neighboring fog locations, cloud data centers, or volunteer end devices. The resource allocation algorithm is responsible for selecting the computation device.
Dedicated Fog Servers -refer to the nodes available at each fog location having significant computing resources with no energy constraints. These servers can accommodate multiple requests at a time. The fog servers share their resource information along with task residence time with gateway nodes. This information is used for computation device selection.
Cloud data centers -This module can handle compute and data-intensive tasks. Moreover, with specific APIs, sensor data can be stored in the data centers. Further, this module is added to support batch processing with no stringent deadlines; whereas, these tasks may require resources for concurrent execution. In such cases, the framework provides an interface to upload those tasks to the cloud where more compute resources are available. Such tasks may include machine learning and artificial intelligence algorithms where the intermediate results are the input for the next layer. In such situations, the cloud data centers perform the required computations and share the results with fog nodes for better intelligent decisions.
Mobility Models & Handover -The mobility is an important aspect of performance degradation in a mobile environment. The fast-moving vehicles or devices can very rapidly disassociate from the network [46]. Therefore, data loss and re-transmissions decrease overall network performance/efficiency. In the proposed framework, devices can adopt different mobility models. The list of mobility modules that exist is Linear, Circular, Gauss Markova, Turtle, etc. Moreover, to handle the impact of various mobility modules, a handover module is inherited to help device association through access points to prevent data loss.

B. PROPOSED RESOURCE SHARING TECHNIQUES
An adoptive resources scheduling algorithm is proposed that inherits properties from two popular algorithms Ant Colony Optimization (ACO) Algorithm and Earliest Deadline First (EDF) Algorithm. In the traditional ACO algorithm, the ants search for food, which is evaluated in terms of quantity and quality before sharing information with others [47]. Similarly, in the proposed algorithm, the broker sends jobs to different nodes, which send an acknowledgment back after completing the job. The broker evaluates the performance of the device in terms of time, the energy consumed, the queue size, and the estimated waiting time for the next job. This helps the broker maintaining the list of available nodes in terms of QoS. The entire process happens asynchronously. The proposed resource manager execution is shown in algorithm 1. This is the main decision module which helps in selecting the most suitable devices for a given task execution request to meet the service quality. The QoS varies depend upon the task stringent deadlines. Therefore, the task having hard deadlines are more suitable to execute on the nearby devices, as it can reduce the communication delay as well as the queuing time. However, as the framework relay on volunteer devices, and in case if no device is available, the system negotiate with the source device for additional time so that the task can execute over fog node. If the source device can tolerate additional delay, the task is scheduled at the fog node. Moreover, the task with flexible or no deadline, the selection modules still check the available devices, their computing power and energy. Thus, device having sufficient energy can be opted for the execution. This selection is performed based on distance (d v using eq. 8) from the source/ requesting device. However, to balanced the workload and to reduce the network traffic, the no deadline tasks are send to cloud data center.
When a task is received at a volunteer/fog/cloud device, it is added to a queue. The queue is ordered using the Earliest Deadline First (EDF) Algorithm [48]. This can help in reducing the re-transmission; on deadline (timer) expiry the system sends a control message to acquire consent to execute the task otherwise, it can simply drop the task. Thus, this way it reduces the retransmission of tasks, and also the failed tasks are dropped from the queue without execution to save time for other tasks to meet the deadline. As illustrated in algorithm 2, an idle device having the queue size zero, the device starts executing the incoming job and sends the result directly to the requesting source device and it also sends an acknowledgment to the broker gateway. This acknowledgment is fruitful in two ways, first, the broker keeps track of each job, means no job will be starved in the queue. Second, the gateway broker ranks this fog node in its list according to the performance which helps in improving the future performance of the system. If the queue size is not zero the incoming job is placed in the queue according to EDF policy. Algorithm 3 gives an overview of the system which describes the processing of the task life cycle.

V. EVALUATION
The proposed simulation framework is evaluated on Ubuntu 16.04 by simulating a smart traffic case scenario.
Case Study -A smart traffic case scenario where roadside units work as fog brokers, dedicated fog nodes are deployed in parking, security cameras, and sensors. The vehicles and pedestrians are used as users and volunteer nodes. In case of any computation requirement, the devices send computation requests to the nearby roadside unit which allocates resources using the list of potential devices. The request size and other networks/ system parameters are given in Table 3. The proposed framework is evaluated in terms of CPU, memory usage, network delay with varying nodes, task acceptance rate, packet drop ratio, execution time, and residual energy. The results are discussed below: Graphical User Interface Drawing Delay -As the proposed framework is developed as an extension of OMNeT++; therefore, it can run in both Graphical User Interface (GUI) and command line interface mode. However, if the GUI based environment is triggered, GUI setup delay is observed due to components drawing on the screen. Figure 3 depicts the setup delay with increasing VOLUME 9, 2021  number of nodes. Therefore, with the increase in the number of nodes, the delay increases, and for 2000 network nodes it reaches about 650 seconds which is too high; however, no sig-  nificant delay has been observed CLI mode. Therefore, CLI mode is recommended for large-scale complex simulation having many devices using heterogeneous mobility models.

Algorithm 3 Gateway Broker-The System Life-cycle
CPU and Memory Usage -In the proposed simulator, each node consists of several sub-modules which include the mobility, application, and energy module. Hence, with the increase in the number of nodes/devices the memory usage and CPU usage increase as well. Figure 4 shows the CPU and memory usage observed with increasing the number of nodes and task generation rate from 200 -800 ms. Figure 4(b) shows the CPU usage with various task generation rate. As the task rate increases, the system CPU usage also increases. Similarly, memory consumption is also compared with increasing nodes and task generation rate. Figure 4(a) shows that more memory resource usage.
Network Delay -The network delay is a core parameter to understand the quality of service offered by a framework.  It also depends on the task generation rate. The framework evaluation is performed in terms of quality of service and task generation rate. Figure 5(a) depicts the network delay with varied quality of service. The figure shows the network delay while executing the task at various layers to meet their service demand. The network delay is proportional to the distance between devices and available data rate [49]. The proposed algorithm tries to meet the service requirement through volunteer devices, local execution at fog nodes, or through the help of cloud data centers. However, if no suitable volunteer node is available, the task service demand is met through execution at the fog node. The network delay increases linearly with the increase in the number of nodes because more nodes requesting compute requests which creates long queues at nearby fog locations and causes network traffic congestion. Figure 5(b) shows the network delay with varying task generation rate (200-800ms).
Efficiency(%) -The efficiency is defined as the ratio of executed and total generated tasks. The efficiency of a proposed framework is compared with xFogSim [23] and the results are shown in Figure 6. Task acceptance ratio means the number of tasks a fog/volunteer node accepts. The drop  tail queue is implemented at every fog node; thus, after the pre-defined threshold, the fog device stops accepting incoming tasks. Thus, on a higher arrival rate, the xFogsim denying the device resource request due to the limited number of resources available at the fog node. Whereas the proposed framework has more resources due to the usage of volunteer nodes, the task service rate is high.
Packet Drop Ratio (%) -Packet drop is an important measure especially in a wireless environment where the low-level devices can send messages without knowing the channel availability. In our framework, the device can send packets after a 200ms gap to avoid channel congestion; however, due to a large number of devices offloading tasks, a bottleneck is developed at the gateway brokers. Thus, due to the limited buffer size, the gateway drop packets. Figure7 show a packet drop ratio with varying number of nodes. Thus, to resolve such issues, macro levels deployment can facilitate better results as discuss in future directions.
System Cost -The system cost is defined as the sum of communication, queueing delay, and execution time. It is an important measure to gauge the performance of the system to meet the quality of service. Fig. 8 represents the system cost at different levels which utilized to meet the QoS. It is observed that the cost is maximum when tasks are uploaded to cloud data centers because of the communication and queueing delay. Whereas, the cost is minimal when jobs are forwarded to volunteer devices. It is due to the device-todevice communication and limited queue size at volunteer devices. Further, the cost at fog nodes is intermediate as it has long queues due to its centralized regional deployment and intermediate communication delay.
Residual Energy - Fig. 9 shows the average residual energy of volunteer computing devices concerning simulation time. The energy of the devices decreases as simulation time passes as it is consumed for task execution. The proposed framework also provides an energy module to observe energy usage due to resource sharing.

VI. AVAILABILITY
The simulator is developed under GNU General Public License, freely available for academics research purpose. It can be accessed from github at https://github.com/rtqayyum/ mFogSim.

VII. CONCLUSION
Fog computing has proven its role for delay-sensitive applications in both domestic and industrial environments. This paper proposed a simulation framework based on a IoT device grid that can act as a fog node to execute the requested tasks. The mobile devices are promoted to fog nodes depending on their energy level and computation capacity. The nodes can move following the pre-defined mobility models. To achieve service quality, an algorithm is proposed which helps in better utilizing the available resources and achieve maximum system throughput. Thus, the framework provides a multilevel resource sharing model. The tasks are allocated based on task deadlines. The evaluation section shows a significant gain in performance by adopting various IoT devices as a work force for the roadside units. However, the proposed work relies on nearby IoT devices; thus, limited availability or access to these devices can effect the performance of the proposed system.
In the future, we are planning to expand this work by incorporating machine learning algorithms to predict device mobility patterns and select the most suitable device for task offloading. Similarly, another way forward is to explore the task dependency issues for the fog computing environment.