A Collaborative Computation and Offloading for Compute-Intensive and Latency-Sensitive Dependency-Aware Tasks in Dew-Enabled Vehicular Fog Computing: A Federated Deep Q-Learning Approach

The demand for vehicular networks is prolifically emerging as it supports advancing in capabilities and qualities of vehicle services. However, this vehicular network cannot solely carry out latency-sensitive and compute-intensive tasks, as the slightest delay may cause any catastrophe. Therefore, fog computing can be a viable solution as an integration to address the aforementioned challenges. Moreover, it complements Cloud computing as it reduces the incurred latency and ingress traffic by shifting the computing resources to the edge of the network. This work investigated task offloading methods in Vehicular Fog Computing (VFC) networks and proposes a Federated learning-supported Deep Q-Learning-based (FedDQL) technique for optimal offloading of tasks in a collaborative computing paradigm. The proposed offloading method in the VFC network performs computations, communications, offloading, and resource utilization considering the latency and energy consumption. The trade-offs between latency and computing and communication constraints were considered in this scenario. The FedDQL scheme was validated for dependent task sets to analyze the efficacy of this method. Finally, the results of extensive simulations provide evidence that the proposed method outperforms others with an average improvement of 49%, 34.3%, 29.2%, 16.2% and 8.21%, respectively.


A Collaborative Computation and Offloading for Compute-Intensive and Latency-Sensitive
Dependency-Aware Tasks in Dew-Enabled Vehicular Fog Computing: A Federated Deep Q-Learning Approach Kaushik Mishra, Member, IEEE, Goluguri N. V. Rajareddy , Umashankar Ghugar, Member, IEEE, Gurpreet Singh Chhabra , and Amir H. Gandomi , Senior Member, IEEE Abstract-The demand for vehicular networks is prolifically emerging as it supports advancing in capabilities and qualities of vehicle services.However, this vehicular network cannot solely carry out latency-sensitive and compute-intensive tasks, as the slightest delay may cause any catastrophe.Therefore, fog computing can be a viable solution as an integration to address the aforementioned challenges.Moreover, it complements Cloud computing as it reduces the incurred latency and ingress traffic by shifting the computing resources to the edge of the network.This work investigated task offloading methods in Vehicular Fog Computing (VFC) networks and proposes a Federated learning-supported Deep Q-Learning-based (FedDQL) technique for optimal offloading of tasks in a collaborative computing paradigm.The proposed offloading method in the VFC network performs computations, communications, offloading, and resource utilization considering the latency and energy consumption.The trade-offs between latency and computing and communication constraints were considered in this scenario.The FedDQL scheme was validated for dependent task sets to analyze the efficacy of this method.Finally, the results of extensive simulations provide evidence that the proposed method outperforms others with an average improvement of 49%, 34.3%, 29.2%, 16.2% and 8.21%, respectively.Index Terms-Deep reinforcement learning, federated learning, mobile fog computing, q-learning, vehicular fog computing, task dependency, task offloading.

I. INTRODUCTION
V EHICULAR networks play a consequential part in smart transportation systems due to the rapid evolvement of the Internet of Things (IoT).These vehicular networks facilitate numerous advanced yet complex applications, such as automatic driving, crash detection, AR&VR-enabled intelligent applications, and other interactive modules for passengers.These applications need intensive computations on resources and interactive communications, which are critical challenges for vehicular networks requiring rich complex services.Besides, the ingress traffic on the road makes the computations/communications difficult with the limited capabilities of vehicles.These applications on vehicles are required to handle latency-sensitive data without delay, for which the network connectivity must be stable and accelerated to handle such tasks within a deterministic span of time.However, the slightest delay in communication between vehicles may cause a catastrophe.Therefore, the vehicular network cannot solely be responsible for these latency-sensitive and computeintensive tasks, thus a high-end computing paradigm-based vehicular network is required.
Cloud computing has been viewed as a viable solution to address these issues for vehicular networks [1], [2].On this Cloud-based vehicular network, tasks are offloaded to the Cloud for computation and storage due to the high computing capabilities of Cloud VMs.However, the existence of the physical gap between the vehicles and the Cloud server results in a significant latency gap, which reduces the performance and efficiency of task offloading.Thus, it requires a decentralized architecture and full-fledged paradigm that reduces the incurred latency and copes with the compute-and time-intensive and latency-sensitive tasks requiring complex requirements in complementary with Cloud computing.
Fog-based vehicular computing (VFC) is a cutting-edge computing paradigm to address the innate loopholes of the Cloud-based vehicular network [3], [4].To tackle the requirements of complex tasks, VFC has been envisioned as a potential solution, providing both on-demand computation and communication resources.Integrating Fog computing with Cloud computing considerably reduces the latency and traffic density as well as improves the users' response time.The computation and communication at/to the Fog computing work in parallel with the services provided at the edge of the network through the Mobile Edge Computing (MEC) or Mobile Fog Computing (MFC) servers equipped in each Road Side Unit (RSU) and Access Points (APs) [5].However, these RSUs have much less capacity in terms of power, storage and computation.The computation of high-intensive tasks causes high energy consumption at RSUs.In the VFC network, the Fog servers are coupled with RSUs or deployed as a separate entity.In this work, we consider the decoupling mode of Fog servers due to the flexibility in the computations of tasks with disparate requirements.The Fog layer encompasses both homogeneous and heterogeneous Fog nodes to cope with compute-intensive and latency-sensitive tasks.The computational capabilities of Fog nodes change depending upon the vehicular specifications.In architecture, each RSU keeps in touch with all the Fog servers and their corresponding Fog nodes.At a given time, one Fog server may be connected to several RSUs, but each RSU is only ever connected to one Fog server.An RSU either computes locally or offloads a request to the Fog server for processing when it receives one.

A. Problem Definition
The VFC imposes several challenges.The first and most challenging issue is unstable network connectivity.Internet connectivity is an indispensable part of vehicular networks needed for transmitting any request.Unstable network connectivity may lead to a catastrophic situation.Therefore, a Dew-enabled Internet of Things is facilitated in this architecture.
The second most prominent challenge is the high consumption of energy by the RSUs [6].A considerable amount of energy is consumed by the RSUs during the offloading of the requests and processing of some requests locally.The incurred consumption is dependent on the quality of the medium connected to the Fog servers and the arrival rate of the requests at the RSUs.Although it processes some requests locally, it also consumes energy due to the disparate specifications and requirements of requests.Hence, there is a need for an optimal association among RSUs and Fog servers to reduce energy consumption.
A third challenge is the dispersion of the loads uniformly across Fog nodes to improve the QoS.Khattak et al. [7] and Kai et al. [8] proposed the integration of the principles of Fog computing with the vehicular ad-hoc network.However, the load balancing among Fog nodes in terms of QoS was not considered.Though the loads are of different configurations, these require to be offloaded uniformly across Fog nodes to prevent overloaded or underloaded conditions.The computation of loads depends on the arrival rate and requirements of each request at the corresponding RSU.The overloaded/underloaded condition arises due to the involvement of the different arrival rates at RSUs.In order to increase QoS (resource usage), the collaboration between RSUs and Fog servers must be established in order to significantly reduce the load imbalance.
The system costs also pose another challenge in these vehicular networks.The communication and computation costs together make the system costs.Earlier, RSUs were powered by renewable energy for the offloading and the MFC servers were generating profits from them.However, since the existing resources have become scarcer due to the rapid ingress of traffic with other challenges nowadays, the vehicles can lease the resources to facilitate the task offloading.

B. Motivation
Deep learning (DL), a subset of the Machine Learning (ML) domain, is a promising solution to address complex problems.AI is an enabler for DL to mimic the learning process.Deep Reinforcement Learning (DRL) combines DL and RL, making use of DL's noncognitive behaviour and RL's ability to make decisions [9].DRL interacts with the vehicles directly and obtains the optimal scheduling/offloading strategy mapping.Based on the existing literature, several works have used DRL to find the optimal offloading decision.For instance, Yao et al. [9] proposed a hybrid resource allocation strategy for VFC using reinforcement learning with heuristic information.Qu et al. [10] proposed a DMRO algorithm integrating DNN with Q-learning and meta-learning approaches to identify the optimal offloading decisions.Ning et al. [11] devised a resource management algorithm using DRL for VEC.To meliorate the effectiveness of vehicular networks (VNs), Maan and Chaba [12] devised a strategy using a Deep Q-network for offloading.To optimize the total utility of VEC, Liu et al. [13] implemented a combined offloading strategy utilizing Q-learning and DRL together.He et al. [14] proposed an offloading method using DRL to improve the Quality of Experience (QoE) for the Internet of Vehicles.However, load balancing was not considered by the aforementioned strategies while allocating the tasks from vehicles, leading to insignificant resource utilization and latency overhead.The traditional ML methods utilize a central server to collect and process the data.However, the central server gets overloaded with computation and communication overheads.Because data privacy is also an indispensable part of data acquisition, the efficiency and accuracy of ML techniques largely depend on the data's size and the central server's capacity, which exhibit challenges in achieving optimal and accurate results with data confidentiality.In order to address the aforementioned issue of traditional ML and DRL techniques, this work proposes a Federated learning-supported Deep Q-learning offloading strategy that facilitates the uploading of data collaboratively into a global model without sharing the raw data [15].
Our work aimed to analyze the task-offloading strategy in two scenarios: (1) considering cooperative and noncooperative MFC servers, and (2) considering task dependency.In the first case, when a vehicle passes through an RSU, the tasks related to the respective vehicle are independently computed by the MFC server associated with that RSU.On the contrary, when many vehicles offload their tasks to the RSU, the RSU chooses whether to compute such tasks locally or transfer them to the next RSU (located in the direction of the moving vehicle) deployed at Fog servers for computation.Hence, the computation is being performed cooperatively.In the second case, we consider dependent tasks for computation.In vehicular networks, vehicles generate a huge amount of tasks with heterogeneous requirements that need parallel execution and, hence, are divided into many sub-tasks.Each sub-task is then processed by different computing nodes based on the offloading strategy to reduce both the computation and response time.Figure 1(a) represents the inter-sub-dependency of a task through a Directed Acyclic Graph (DAG), where the task is divided into seven sub-tasks.Figure 1(b) shows the fine-grained task scheduling in the Vehicle-VFC-Cloud collaborative framework.

C. Contribution
As mentioned, this paper proposes a Dew-enabled Federated learning-supported Deep Q-learning-based (FedDQL) offloading method for compute-intensive and latency-sensitive dependent tasks for a collaborative DIoT-Edge-Fog-Cloud framework.This strategy collaboratively optimizes the latency and computations for its intended tasks by reducing the energy consumption of MFC servers.Moreover, the computation loads are uniformly distributed across Fog nodes to prevent load imbalance and, thus, maximize resource utilization.The Fog nodes in the Fog layer are grouped into clusters, each of which contains a homogeneous Fog server, heterogeneous Fog server, Fog nodes, and numerous RSUs.In addition, each cluster has a DQL agent to train the DQL to determine the offloading decisions.A collaborative framework is proposed for tasks offloading in Vehicular Fog Computing.Intermediate nodes, such as routers, gateways, or switches, are deployed on the edge of the network along with a classifier to classify the incoming requests and offload them to the target layers for computation.The Fuzzy classifier is a classifier that determines the target layer for offloading based on the tasks' requirements, including network bandwidth, size in MI, CPU, resource utilization, etc.
The contributions of this paper are delineated as follows: • Modelling the framework with the principles of Dew computing in order to enable the Internet of Vehicles for uninterrupted task offloading; • Proposing a collaborative Dew-enabled IoT-Edge-Fog-Cloud computing framework for efficient offloading of tasks, where a DAG depicts the interdependencies among sub-tasks; • Proposing a Federated learning-supported Deep Q-learning algorithm for dependency-aware task offloading; • Considering load-balancing, resource utilization, energy consumption, response time (latency), and deadline as QoS metrics to appraise the efficacy of the implemented method; and • Conducting an empirical assessment of the proposed offloading strategy with other existing methods.The rest of the paper is organized as follows.Section II reviews the related research.Section III describes the computational models and problem formulation for computeintensive and latency-sensitive dependency-aware task offloading.The proposed Federated learning-supported Deep Qlearning offloading strategy is elucidated in Section IV.Section V provides an empirical assessment and analysis of the considered QoS parameters for the proposed strategy.Finally, Section VI concludes the paper and highlights future research directions.

II. RELATED RESEARCH
The VANET model has been extended to Vehicular Fog Computing (VFC) by enabling the core principles of Fog computing for the effective offloading of latency-sensitive tasks to compute-intensive vehicles.It facilitates reducing energy consumption, response time, load balancing, and latency while improving resource utilization and, thus, performance.The related research boils down to two primary aspects: (1) Fog computing in vehicular networks, and (2) deep learning for task offloading.The existing works for both of these aspects are briefly summarized as follows.

A. Fog-Assisted Vehicular Networks
A great deal of work has addressed the issues related to vehicular networks using the appealing characteristics of Fog computing in collaboration with the Cloud [16], [17], [18], [19].For instance, to reduce the total response time of delaysensitive tasks, the authors of [16] proposed a heuristic-based greedy scheduling algorithm for a three-layered VFC architecture for offloading tasks.Next, a Fog-assisted vehicular computing framework is presented in [17], which is utilized as a testbed for controlling the ingress traffic on the roads.The authors stated that the computations are performed by the RSUs deployed along the road with the assistance of mobile vehicles to meet the demand of the ever-increasing growth of compute-intensive tasks by smart vehicles.To identify congestion among vehicles, [18] proposed a Fog-enabled framework to process the sensed data locally by optimizing the communication.To preserve the privacy of VFC, the authors of [19] introduced a real-time framework called GALAXY supported by Federated learning without the aid of the Cloud.

B. Deep Learning for Task Offloading
In an aim to reduce the transmission power and latency, the authors of [20] devised a collaborative optimization technique to allocate tasks into Fog nodes in an IoT-Fog-Cloud architecture.For efficient resource management, [21] proposed a two-fold technique for allocating tasks and managing vehicular resources optimally in a VFC.In the latter work, contract theory and two-side matching games were formulated for Authorized licensed use limited to the terms of the applicable license agreement with IEEE.Restrictions apply.
resource management and task allocation, respectively.Using an actor-critic-based DRL algorithm, the authors of [22] devised an offloading algorithm considering priority in VFC by predicting the dynamic pricing of vehicles.In [4], the authors developed a resource allocation algorithm for parked and slowmoving vehicles in VFC, aiming to reduce parked vehicle service latency.The authors used a combined recurrent neural network (RNN) with DRL and, later, introduced a heuristic strategy to expedite the convergence rate of RL algorithms.To lessen the VM migration and reduce the power consumption during migration, the authors of [23] devised a mechanism called EnTruVe (ENergy and TRUst-aware VM allocation in VEhicular Fog Computing).In addition, an analytical hierarchy process was used to determine an optimal VM based on trust and latency, providing a large pool of vehicular fogs to satisfy the requirements of VMs.In order to minimize the computation-to-communication cost and latency, the authors of [24] proposed a Fuzzy RL-based offloading algorithm that elevates the learning process and maximizes the long-term reward over the Q-learning algorithm.The authors of [25] introduced a dynamic clustering-enabled clustering-based load balancing approach.Moreover, the technique considers the speed, direction, and position of moving vehicles to form groups for making a pool of computing resources.In addition, it also employs a capacity-based load-balancing approach for performing the distribution of loads between vehicles and among the cluster of vehicles.To decide the target layers for offloading, the authors of [26] proposed a Federated learningsupported DRL mechanism for VFC, which learns through the learning model among Fog nodes and vehicles, resulting in fast convergence.Network overhead is also significantly reduced due to the implementation of Federated learning, and thus, the privacy of the users' data is also preserved.

III. SYSTEM MODELS AND PROBLEM FORMULATION
Initially, this section introduces a dew-enabled VFC network model for task offloading.Next, different system models, including the vehicular task, communication, and computation models, are elaborated.Furthermore, the problem formulation with key objectives is discussed.

A. Dew-Enabled VFC Network Model
The proposed dew-enabled Vehicular Fog Computing model, a collaborative task offloading network/communication framework, is illustrated in Fig. 2. The model consists of five layers: the bottom IoT layer, Dew layer, Edge layer, Fog layer, followed by the Cloud layer.The roles and significance of each layer in task offloading in a vehicular network are described as follows.
At the ground layer, the IoT layer consists of numerous Internet-enabled physical devices (vehicles), which generate requests of disparate specifications.Vehicles also contain local processing units.All communications take place through a stable Internet connection.However, a stable connection around the clock seems impractical, and any catastrophe may arise due to unstable connectivity.Therefore, the core principles Vehicles with Internet connectivity transmit requests to the Cloud or Fog layer to be processed via the Edge layer.The Edge layer is made up of plenty of intermediary nodes, including routers, switches, or gateways, that are used to direct the requested packets to the following layer immediately.Additionally, it uses a Fuzzy logic classifier to categorize the tasks in order to identify the target offloading layers based on the requirements of each task.However, this layer exhibits some challenges in terms of security, cost, connectivity, scalability and complexity [8], [9], [11].This concept is theoretical and close to impossible in implementation.This layer is a replica of how the computations can be done on the edge of the network via intermediate nodes.The requests are then sent to the Cloud or Fog layer, depending on which one will handle them.The Fog layer is an amalgamation of different Fog nodes with homogeneous and heterogeneous resources.These Fog nodes are computing nodes in terms of virtual machines (VMs) to process requests either collaboratively or locally in this layer.Furthermore, these computing nodes are clustered together based on the computing capacity into two groups: Collaborative Fog Nodes (CFNs), which is a collection of both homogenous and heterogeneous resources that are able to handle tasks (or requests) with disparate specifications; and Local Fog Nodes (LFNs), which is a combination of only homogeneous resources that is capable of handling tasks with minimum completion time or requirements.Moreover, this layer also deploys RSUs and MFC servers to communicate among vehicles.Each RSU is embedded with an MFC server and gathers service requests from vehicles within its range while each MFC server processes tasks and subtasks through offloading.Here, the nodes are geographically distributed facilitating the execution of the tasks in a distributed way to leverage the centralized architecture of the Cloud datacentre.The tasks with high requirements or tasks taking longer to execute in the Fog layer are offloaded to the Cloud for processing and storage.The Cloud datacentre encompasses the physical machines (or hosts) and VMs out of hosts through virtualization technology.The VMs are the computing entities for enabling the scheduling and execution of high-end tasks.The centralized architecture of the Cloud datacentre results in high response time and latency overhead.The Fog layer is introduced in the proposed framework to minimize these issues.
We assume that the tasks are dependent in nature and, hence, partitioned into subtasks.Furthermore, we assume that N compute-intensive and latency-sensitive tasks are connected with each vehicle and will be scheduled and computed within a deterministic completion time.The number of RSUs is represented as R = {1, 2, 3, . . ., r , . . ., R}, and the group of vehicles is referred to as V = {1, 2, 3, . . ., v , . . ., V }.The variable T v n refers to the n th task of the vehicle v.

B. Vehicular Task Model
This section introduces dependency-aware task offloading for the vehicular task model.Interdependency among tasks is considered as the tasks are not atomic.In other words, when a task is broken down into subtasks, the processing of each subtask is dependent upon its previous subtask's execution.The processing of each subtask is based on the requirements of each subtask and is offloaded to the respective target layer for processing (locally (vehicle) or via Fog or Cloud).The subtasks of a task are represented through a Directed Acyclic Graph (DAG) and modelled as G = (τ, ε), where τ is the array of subtasks and ε is the pair of directed edges depicting interdependencies between two subtasks.Let I = |τ | represent the total set of subtasks of a task T v n .In DAG, a vertex T v n,i denotes the i th subtask of the n th task of the vehicle v, and a directed edge (T v n,i , T v n,j ) indicates the interdependency between the i th subtask and j th subtask of T v n .The j th subtask can be executed only if the i th subtask has been completed, where i , j ∈ I .As shown in Fig. 1, the task , and T 3 2,7 of the form T v n, i;i=1,2,...,7 .The subtask T 3 2, 1 is the entry subtask and executes first, and T 3  2,7 is the exit subtask and executes at the end until all its predecessor subtasks have been executed.Each subtask T v n,i is defined with four attributes, i.e., , where L v n,i denotes the length of the subtask in MI, D v n,i is the latency rate of the subtask, d v n,i is the deadline (hard or soft or firm) of the subtask, and C v n,i is the number of resources required for computation to execute T v n,i .In our approach, a task can be categorized into three classes, namely crucial, high-end, and low-end tasks.We assume that the crucial tasks are those which contain some crucial information about the vehicle and, hence, can be computed in the vehicle itself.High-end tasks are associated with some priority and deadline.If the high-end tasks do not meet the deadline, the outcome must not be considered or the task is considered as failed.Low-priority tasks also have some priority and deadlines.Unlike high-priority tasks, if the low-priority tasks do not meet the deadline, the outcome is still in use, but the outcome will not be considered if the execution time increases.If the execution of a task is longer than usual, then it is forwarded to the upper layer for processing.However, the vehicles must ensure the execution of high-priority tasks over low-priority tasks.A utility in terms of reward or penalty is associated with both high-priority and low-priority tasks upon meeting or missing deadlines.For the high-priority task, a log function is used to define the utility of a subtask given as: where t Cn denotes the completion time of a subtask T v n,i ; and −Γ H implies a penalty for not being able to meet the deadline.
For the low-priority task, if the completion time of the executing subtask is less than the defined deadline, then the utility is a reward.Otherwise, the utility is a penalty and expressed as: where Γ L defines a positive reward (constant); and c is a constant and greater than zero.
In order to offload a subtask to any other vehicle (call it a service vehicle), the task vehicle has to pay a unit price denoted as ρ n .Hence, the computation size of the subtask (C n ) can be estimated as C n = f n t n , where f n is the frequency assigned to the service vehicle, and t n denotes the computing time of the subtask.Finally, the utility of the task vehicle can be expressed as: where 1(.) is an indicator function [28].and T H and T L denote the high-priority and low-priority subtasks, respectively.
Authorized licensed use limited to the terms of the applicable license agreement with IEEE.Restrictions apply.

C. Classification of Offloading Layers Using Fuzzy Logic
Fuzzy logic is used to determine the target offloading layers for the tasks generated by vehicles.It is implemented to minimize the latency incurred while executing high-end and low-end tasks at one layer.Specifically, Fuzzy logic offloads different groups of tasks requiring disparate computational capabilities of resources to different target layers.It prevents starvation and aging.By this method, if the execution of a task is taking longer than usual, the respective task is forwarded to the next adjacent layer for execution.It is manifested through three modules: Fuzzy inputs, Fuzzification, and Defuzzification.Fuzzy inputs are the essential parameters of the Fuzzy logic model that draw the inferences, which include Tasks size (MI), Network Bandwidth (Mbps), Latency-sensitivity, and Deadline-constrained.These inputs are characterized by High (h), Medium (m), and Low (l) lexical parameters representing the heterogeneity and dynamicity of tasks.Fuzzification is the process of drawing inferences based on Fuzzy inputs with the aid of an inference engine.Fuzzy Knowledge Base is a repository consisting of inferences in terms of Fuzzy rules.Fuzzy inputs are processed based on the defined membership functions.Defuzzification is a process of transforming the Fuzzy inferences to some consolidated values based on membership functions.Table I delineates the required output (target offloading layers) based on Fuzzy inputs.

D. Vehicular Communication Model
This section describes the communications and interactions among different entities for smooth offloading.The interactions refer to the communication model of vehicles to RSUs, from RSUs to the Cloud server, and from vehicles to the Cloud.The communications among these entities are illustrated as follows.
Vehicles to RSUs: The Internet-enabled vehicles (IoV) communicate with RSUs through wireless communication based on Frequency-Division Multiple Access (FDMA).To transmit a byte of request, a vehicle v requires Tr r v amount of data transmission rate to an RSU r, expressed in Eq. ( 4) [27]: where h r v is the allocated sub-channels to the vehicle v; bw is the allocated network bandwidth for each sub-channel; Tp v is the power to transmit each byte from the v th vehicle; β v is the gain of the channel; and ρ −2 is the noise in the surrounding environment.
RSUs to Cloud Server: An optical fibre-based wired connection is used to transmit tasks from RSUs to the Cloud server.If the MFC server fails, tasks or subtasks are migrated to the Cloud server for computation.Some transmission latency occurs while offloading to the Cloud.The transmission latency and acknowledgement latency are equivalent between the MFC server and Cloud.While the transmission latency is independent of the lengths of the tasks, it results due to the presence of the physical gap between the MFC and Cloud.Nonetheless, the incurred latency, in this case, is lesser than the direct offloading of tasks from vehicles to the Cloud server.Therefore, we denote the round-trip time (τ Cloud r ) for transmitting data from the RSU to the Cloud and get the ack in Eq. ( 5): where D Cloud off (r ) denotes the transmission latency incurred while offloading a request from RSU r to the Cloud, and D r ack (Cloud) is the latency incurred for sending an ack from the Cloud to the RSU r.
Vehicles to Cloud server: The high-end tasks are directly offloaded to the Cloud servers for computation, thus requiring high-end specifications.Therefore, the communication involves the transmission latency of sending a request from a vehicle v to the Cloud server and is expressed in Eq. ( 6): where L i denotes the length of the i th request; and h Cloud v is the allocated sub-channels to the vehicle v.

E. Computation Model
This section explains the different computation models in this vehicular Fog computing architecture, which includes the energy consumption model, latency-aware computation model, and pricing model.Each model is elucidated below.
1) Energy Consumption Model: The total energy consumed at time t when a vehicle transmits a request either to the Fog layer or the Cloud layer via RSU is expressed in Eq. ( 7): The energy consumed at time t at the RSU r (Ec r i (t)) while propagating the request from the vehicles is estimated in Eq. ( 8): where L i is the length of the i th request; Ec r nw (i , t) denotes the energy consumed to transmit each byte of request from a vehicle v to the RSU r via the network; h i is the transmission gain of the channel; P r idle and P r max are the energy consumed during the idle and active states of the RSU r, respectively; Cap r idle and Cap r max are the capacity of the RSU r during the idle and active states, respectively; and Util r is the utilization of the RSU r.
Next, the task is either offloaded to the Fog or Cloud for computation.Here, the energy is consumed while processing and storing the requests as well as sending the respective acknowledgement.Hence, the energy is mathematically expressed as follows: The energy consumed while processing and storing the request either at Fog node or Cloud VM, Ec P &S i , is expressed in Eq. (10): where δ is the ratio ( T active T total ) of the active time to total time of the j th Fog node or Cloud VM; (1 − δ) denotes the elapsed time for the j th Fog node or Cloud VM; P active j and P idle j denote the power consumption of the j th Fog node or Cloud VM during the idle and active states, respectively; St(j, t) is the service time of the j th Fog node or Cloud VM; nT(j, t) is the number of tasks associated with the j th Fog node or Cloud VM; and β(= T sent req Treq ) is the ratio of the total request sent to the total request offloaded to the Cloud from Fog nodes upon failure.
Ec Cloud i is the energy consumed by the j th Cloud VM when any Fog node fails to execute, as computed in Eq. ( 12): Ec ack is the energy consumed while sending the acknowledgement back to RSU or the corresponding Fog node and is expressed in Eq. ( 13): 2) Latency-aware Model: A task's subtasks can be carried out locally (on the vehicle itself) or offloaded to a Cloud server or an MFC server at the Fog layer for processing.At various phases of processing and offloading, variable rates of latency might occur.In this instance, the processing times of the subtasks at the vehicles, MFC server, and Cloud server can be used to determine the latency.
Computing on vehicles: When the tasks are classified and offloaded to the target layers for computation, tasks with their subtasks arrive at the queue of the vehicle for computation.We assume that all the tasks in the queue are in FIFO order and the vehicle processes one subtask at a time.That means, while a subtask is in execution, other subtasks must wait in the queue.This processing involves both processing time and waiting time in the queue.Hence, the total latency incurred while computing on vehicles for a subtask T v n,i is given by Eq. ( 14): request(n,i) (14) where is the waiting time of a subtask T v n,i ; C v n,i is the number of computing resources needed to execute the subtask T v n,i ; c v is the computation capability for the vehicle v; and T request(n,i) are the starting and requested times of a subtask T v n,i , respectively.Computing on MFC server: The latency rate at the MFC server is the sum of the transmission delay from the vehicle to the MFC server via RSU, processing delay, waiting delay of the subtasks on the MFC server to be scheduled, acknowledgement delay back to the vehicle, and the offloading latency to the Cloud upon failure.These factors are formulated as follows: First, the transmission delay in propagating a request i to the respective Fog node is estimated as: where T nw L denotes the transmission delay caused by propagating each byte of request i from the vehicle v to an MFC server fn j ; j = 1, 2, . . ., m.
Second, the latency caused by processing the request i on an MFC server is denoted as: Third, transmitting an acknowledgement to the respective vehicle v causes some latency and is denoted as: Next, the waiting latency at the queue of the MFC server is similar to the local waiting delay and is expressed as follows: Finally, the transmission delay caused to offload of a request i to one of the Cloud servers upon failure is expressed as Authorized licensed use limited to the terms of the applicable license agreement with IEEE.Restrictions apply. follows: Here, the transmission delay states that when a request is sent from an IoT device to a Cloud server, the time it takes for the request to reach the server (transmission delay) is equivalent to the time it takes for the acknowledgement (response) to travel back to the IoT device.Therefore, the transmission delay is doubled because it includes both the time for the request to reach the server and the time for the response to reach the IoT device.So, the total transmission delay is the original delay multiplied by two.
By combining Eqs. ( 15)-( 19), we get the total latency caused at the MFC server in Eq. ( 20): Computing on Cloud Server: The subtask T v n,i is migrated to the Cloud server after being offloaded from the vehicle v to the RSU r for processing.The transmission latency includes transmission delay, processing delay at the Cloud server, and acknowledgement delay, which are analogous to the MFC server.Hence, the total latency caused by the Cloud server is defined as follows: 3) Pricing Model: In VFC, the low-end tasks are computed on the vehicle itself, and the high-end and medium-end tasks get scheduled on the Cloud server and Fog server due to the lack of computational capabilities of the vehicles.Hence, a fair allocation of resources should adhere to the computation of high-end tasks to meet their deadline on a priority basis.Let's consider T n number of high-end tasks that are generated from the vehicles V v .The deadline and computation size of a task/subtask are referred to as d i and L i , respectively.The minimum frequency required for the local subtasks is denoted as Upon the availability of computing resources, a vehicle might deny offloading the requests.Then, the frequency is computed as Here, ϑ is used to denote the ratio of minimum and maximum frequencies to compute the subtasks locally, and the range of ϑ is [ϑ v , 1].Therefore, we denote the total utilization of local subtasks as [22]: In one scenario, a vehicle migrates its subtasks to a passing vehicle for computation due to the lack of computational capabilities and allocates frequency f n for the migrated subtask T v n, i .Therefore, the utilization of the migrated local subtasks on a new vehicle is denoted as Util v local (ϑ v ), and the cost (ρ n ) paid for the subtask T v n, i should meet the requirement as . This is compensation for the loss of the utilization of the local subtask.The frequency devoted to the offloaded subtask is estimated as where ϑ v can be estimated from the above requirement condition [22].Consequently, if there is a need for computing resources in order to compute the offloaded subtasks, then the cost will be higher for the respective vehicle v.

F. Problem Formulation
The objective functions can be formulated as: subject to the following constraints: C6 : Eq. ( 23) indicates two key objectives: (1) reducing the overall computational latency ratio across the computing layers, and (2) minimizing the average energy consumption across RSUs.Eq. ( 24) implies that the average utilization of offloading subtasks should be maximized.
The constraint C1 in Eq. ( 25a) is the addition of weights (β and α) of two sub-objectives (T L v n, i (t) and E c (t)) stated in Eq. (23).Eq. (25b) represents the total computational latency incurred while transmitting requests from the RSU to Fog, RSU to Cloud, and then Fog to Cloud.In Eq. (25c), X i, j (t) is a binary variable that states the assignment of the subtasks to the desired MFC servers for computation.Eq. (25d) states that the RSU j can be linked with at least one Fog server m, but a Fog server m can be linked with more than zero RSUs at a given instant.In Eq. (25e), Comp v , m and Comm v , m are the allocated computation and communication resources, respectively, and should not exceed the maximum computation and communication capabilities of MFC servers.The constraint C7 guarantees a positive cost value within the range of the maximum value, and ϑ v denotes the current value of ϑ in V v while offloading a task.The constraint C8 denotes the availability of the selected vehicle for the subtask offloading.
This section formalized the task offloading problem in VFC.This optimization problem aims to reduce the total computational latency and mean energy consumption while maximizing the total utilization of offloading subtasks in terms of cost subject to the constraints of communication and computing resources, the nature of the association between the RSUs and Fog servers, the binary mapping of subtasks to the compatible Fog servers or Cloud servers, and the availability of the selected vehicle to offload the subtask when a vehicle fails to compute.In a dynamic context (unpredictable requests with disparate requirements) where the level of collaboration between RSUs and Fog servers/Cloud servers fluctuates, it is challenging to optimize the objective functions in Equations 23 and 24 using conventional optimization approaches like heuristics or metaheuristics.In other words, it is not easy to find the best solutions for the objectives given the constantly changing level of collaboration between the servers and RSUs due to the unpredictable demands.Hence, the next section proposes a Federated learning-supported Deep Q-learning offloading (FedDQL) strategy to optimize the aforementioned objective functions.

IV. TASK OFFLOADING ALGORITHM: A FEDERATED LEARNING (FL)-SUPPORTED DEEP Q-LEARNING (FEDDQL) APPROACH
This section first discusses the conventional Deep Qlearning neural networks for identifying a dynamic vehicular offloading in the Fog paradigm.Afterwards, a Federated Learning (FL)-supported Deep Q-learning strategy for global offloading is presented.In the proposed offloading strategy, the conventional Q-learning is amalgamated with deep neural networks as the proposed DQN strategy.In this network, an agent acts by learning from the environment and optimizes the sum of rewards.This environment, in terms of the VFC environment, can be formulated as a Markov Decision Process expressed as env : {S(t), Ac(t), Re(t), P(t)}, where S(t) denotes the state space of the environment at time t, Ac(t) implies the action at time t performed by the DQN agent, Re(t) expresses the reward space an agent gets, and P(t) is the probability of transition.Here, an agent indicates the vehicle associated with the number of RSUs and Fog/Cloud servers in the Vehicle-Fog-Cloud environment.Each terminology used in the environment is illustrated as follows.
State: A state space of an environment is comprised of computational models in terms of mean energy consumption of RSUs and the computational latency incurred while transmitting request i from a vehicle v and is defined as [11]: where Ec(i, t) is the energy consumption by the i th RSU at time t; and D(j, t) is the computational latency incurred by the j th Fog server at time t.Action: An action space comprises an action in terms of offloading the requests by an agent.Hence, the action Ac(t) of a subtask i of the task T v n, i at time t is given by [11]: where X i, j (t) is the mapping of subtask i to a compatible the j th Fog/Cloud server; and Alloc comp (t) and Alloc comm (t) are the allocated computation and communication resources for the subtask i of the task T v n, i , respectively.Reward: Based on the execution of the action on the state defined in the formulated environment, a reward is facilitated.The reward Re(t) is the negation of the weighted total of both average energy consumption and computational latency at time t given as [11]: An agent must select the action Ac(t) as an offloading of the subtask i at state S(t) to get the optimal reward while minimizing the total energy consumption across RSUs and the computational latency.In the next sections, the conventional Q-learning approach is demonstrated, followed by the proposed FedDQL offloading strategy.

A. Standard Q-Learning Approach
The Q-learning approach is used to identify the optimal decision-making policy through a function Q{S (t) → Ac(t)}.It is a guiding force for the agent in the environment to maximize the reward and is defined through a deterministic Bellman equation [29] as follows [13]: where η is the learning rate; and Υ denotes the discount rate.
The Q-learning approach first initializes the Q-value to zero for all periods.The agent gains the state space S(t) information from the environment (RSUs) at time t.According to the state, the agent selects an action Ac(t) from the Q(S (t −1), Ac(t −1) ) using an − greedy policy.Using this policy, the agent identifies the best action with (1 − ) probability rate and random action with probability rate.Further, a Q-value is updated in each iteration, and the mean reward is calculated afterwards.

B. Deep Q-Learning-Based Neural Network (DQNN) Approach
The Q-learning method is primarily used for a small set of state and action spaces.Since the Vehicular Fog Computing environment is dynamic in nature, the Q-learning approach is not suitable for finding the optimal reward function.Therefore, a Deep Neural Network is incorporated into the Q-learning approach with a parameter set θ to deal with the dynamic nature of tasks for optimal results.In DNN, θ is used to map the input state to action.In order to reduce the loss, a target layer is added to the primary neural network for network stabilization.The loss function of this neural network is expressed in Eq. ( 31) [31]: Authorized licensed use limited to the terms of the applicable license agreement with IEEE.Restrictions apply.
where O DQNN denotes the output of the DQNN and is expressed in Eq. (32): Q(S (t + 1), Ac(t + 1)) (32) For network stabilization, the transition table is updated with tuples, such as {S(t ), Ac(t), Re(t + 1), S(t + 1)}.The DQNN works as follows: the parameters of the primary and target layers are initialized.The DQNN agent gains the state space information from all the RSUs, then identifies an action space using an -greedy policy.Next, the transition table is updated with the reward and state space data.Furthermore, the weights are updated, and the loss function specified in Eq. ( 31) is diminished using a gradient descent approach.The DQNN duplicates the parameters (θ) of a target layer from the primary network at the end of a specific phase.

C. FedDQL Approach
Federated learning (FL) is a cutting-edge concept that considers all the components to design a global model that does not need to share the original data.Herein, the core features of FL were incorporated with DQNN to develop the FedDQL global model, which has two primary advantages: (a) privacy preservation in terms of sharing confidential data, and (b) partial participation of the Fog servers that are energy-deficient.The Fog geographic region is scattered with clusters of Fog zones consisting of one or more RSUs and homogeneous and heterogeneous Fog servers.Each Fog zone has a DQNN agent to determine the offloading scheme with the DQNN approach.
In FL architecture, which is a distributed machine learning approach, there is a single model that is responsible for calculating and setting all the global metrics.This model is called the "unified model," and it acts as a central point of coordination for the distributed learning process.In the FedDQL model, the DQNN agent of each Fog zone downloads the global metrics from the unified model and trains their local model accordingly.After training, the metrics are updated and then forwarded to the unified model.Afterwards, the unified model aggregates all the local metrics and forms the global metrics.In this model, all the non-involving Fog zones merely download the unified model to keep an update.
This work considers M set of involving Fog zones of size K as K = P |Z (f )|, where P denotes the involvement factor, and Z(f ) is the clusters of Fog zones.The involved Fog zones are identified according to their uplink transmission cost [22], which is calculated as follows: where T m P denotes the power to transmit the local metrics by the m th Fog zone of size l; and A is the rate taken to transmit by the m th Fog zone of size l and is estimated as: where Bw is the channel's bandwidth; h is the channel gain; and In step 1, the array of Fog zones M and the series of involving Fog zones are initialized.The cost of transmission by each Fog zone is estimated using Eq.(33).Afterwards, each Fog zone is sorted in ascending order based on their cost of transmission m.Step 4 identifies the first k number of Fog zones in FL, which involves a set of F rounds.In each epoch, each Fog zone updates their local parameter lists and broadcasts it to the unified model to form a global model in steps 5.1 and 5.2.In step 5.3, the unified model obtains a global model (θ f +1 ) by performing aggregation at each round f of the local parameter (θ f m ), which is expressed as [15]: where D m denotes the size of the local buffer of the m th Fog zone; and D implies the total size of data estimated as D = m∈M D m .

V. EXPERIMENTAL ASSESSMENT AND DISCUSSIONS
This section describes the extensive simulations that were performed on a real-world benchmark dataset for different conflicting scheduling parameters to assess the proposed strategy.The simulation parameters were first delineated, then a convergence analysis of the proposed technique was carried out.Afterwards, a comparative performance analysis was achieved for the considered scheduling parameters against some existing works.

A. Setting of the Simulation Environment
iFogSim over CloudSim is used as a simulator to instantiate a VFC simulation environment.This environment encompasses 10 Internet-enabled Vehicles, 3000 dynamic tasks, 296 Fog nodes/VMs, and 16 RSUs.There are multiple MFC servers in this architecture, and each MFC server is connected Authorized licensed use limited to the terms of the applicable license agreement with IEEE.Restrictions apply.to a single Road Side Unit (RSU), which is a device used in vehicular communication systems.Additionally, each of these servers has many cores, which are individual processing units within the server that can execute tasks simultaneously.The lengths of a series of tasks range from 0-15000 MIs with the size of each task ranging from a set of {30, 35, 40, 45, 50, 60} MB.Each task requires the computation resource requirements, which are arbitrarily allocated from a set of {0.6, 0.8, 1.0, 1.2, 1.4} Gigacycle/s.Each task is arbitrarily subdivided into 6-10 subtasks.The transmission power is dependent on the hardware used in the system [35].In the IoT-Edge-Fog-Cloud environment, the transmission power of an IoT device can be dependent on various hardware components, such as the transceiver, amplifier, power supply, and processor.The efficiency of all these components impacts the transmission power of IoT devices.Table II lists the other simulated parameters used in the experiment, and the technical specifications of Fog nodes/VMs are presented in Table III.
A real-world dataset [34] was utilized to assess the performance.The dataset has an ETC matrix containing a number of tasks and machines with a corresponding computation ratio.This dataset has twelve instances in the form of u_x _tm with respect to uniformity in data (u), consistency (X) among data, and heterogeneity of tasks (t) and machines (m).The consistency among data was further classified into three groups: consistent (c), inconsistent (i), and semi-consistent (s).Likewise, the tasks and machines heterogeneity is categorized as high (h) or low (l).As a result, the twelve instances were formed by considering the aforementioned criteria.In addition, a variable number of tasks and machines was considered and formed into three groups: Group 1 (1000 × 96), Group 2 (2000 × 196) and Group 3 (3000 × 294) in the structure (t × m).

B. Convergence Analysis
To demonstrate the convergence analysis of FedDQL over other baseline algorithms, a convergence of the proposed  method was analyzed with different learning rates, and then the convergence rate of other algorithms was analyzed.
The convergence of FedDQL was analyzed with disparate learning rates ranging from 0.01 to 0.09, as depicted in Figure 3.It is evident that the most suitable learning rate is 0.01 with a mean reward of 2025, while the worst learning rate is 0.08 with a mean reward of 2003.Therefore, 0.01 was considered in the simulation.
Figure 4 depicts the convergence rate of FedDQL and other compared algorithms.It can be observed that FedDQL converges faster due to the consideration of the dynamic behaviour of tasks and interdependencies among them.The optimal mapping of tasks into Fog servers or Cloud VMs results in fine-grained offloading.

C. Performance Analysis of Scheduling Metrics
Here, the performance of the FedDQL is analyzed and compared for different scheduling metrics, such as service time (ms), average utilization rate (%), mean energy consumption (KJ), and mean latency rate (ms).The proposed FedDQL is compared with other offloading strategies [23], [30], [31], [32], [33] to gauge its efficacy.
1) Performance Analysis of Service Rate: For latencysensitive applications, the user's request must be serviced in a deterministic time and, hence, plays an indispensable part in this computing paradigm.Figure 5 shows a comparative analysis of the obtained results for service rate in three different sets (tasks × machines).The dew-enabled architecture and Deep Q-learning approach helped minimise the proposed framework's service time.Comparatively, the other approaches Authorized licensed use limited to the terms of the applicable license agreement with IEEE.Restrictions apply.do not consider the decomposition of tasks and classification of the target layers for offloading.Identifying different offloading layers has expedited the execution process and, thus, resulted in reduced service for each user's request.The FedDQL outperforms the other models [23], [30], [31], [32], [33] with an improvement of 49%, 23.2%, 22.6%, 14.7% and 3.23% on average.
2) Performance Analysis of Average Utilization: Resource utilization is a significant part of offloading requests for resources.This factor contributes an equal amount with load balancing in meliorating the performance of any offloading algorithm.The use of the Deep Q-learning network helps to predict the load and assign the requests according to each Fog server, improving the utilization of all the underlying resources considerably.Figure 6 depicts a comparative analysis of the obtained results for the average degree of utilization for three different sets.Nevertheless, all other approaches do not consider load balancing.The FedDQL outperforms the other models [23], [30], [31], [32], [33] with an improvement of 51%, 36.8%,32.4%, 27.7% and 9.33% on average for three sets.
3) Performance Analysis of Latency Rate: Latency, which is caused by the high computational time or delay in transmission and propagation, is a pivotal element for latency-sensitive applications in computing paradigms.Due to the implementation of Fog servers between the Internet-enabled Vehicles and Cloud layer, the latency rate has been notably reduced.Moreover, classifying tasks and determining the offloading layers also minimise latency across computing layers.The latency rate significance when there is an number of tasks in a dynamic environment.Figure 7 presents a comparative of the obtained for latency rate.It obvious that the suggested method presents a remarkable improvement over the other models for different specifications.The FedDQL outperforms the other models [23], [30], [31], [32], [33] with an improvement of 15%, 9.4%, 9.9%, 14.7% and 2.33% on average for three sets.As the number of requests increase, the proposed method shows notable results over [32].For a smaller set of requests, both of these (proposed and [32]) perform approximately.

4) Performance Analysis of Average Energy Consumption:
The degree of energy consumption is a challenging factor for any datacentre.The consumption of energy depends on various factors, such as the specifications of the Fog servers/datacentre, computational capabilities, resource-constrained, size of the tasks, etc.In our approach, the energy consumption is evaluated for processing and storing each task on a resource, transmitting and acknowledging the requests from/to the computing layers.Efficient utilization of resources could minimize the degree of utilization.Figure 8 presents the QoS analysis of the degree of energy consumption for an increasing number of tasks for the proposed method.It is apparent that the suggested method reduces energy consumption more efficiently than the other methods with different degrees of Authorized licensed use limited to the terms of the applicable license agreement with IEEE.Restrictions apply.

VI. CONCLUSION AND FUTURE STUDY
This paper proposes a noble offloading method for compute-intensive and latency-sensitive dependency-aware tasks in the Dew-enabled collaborative computing framework.The dependency-aware tasks are depicted through a DAG, where the interdependencies among tasks are also modelled.Moreover, a Federated learning-supported Deep Q-learning (FedDQL)-based offloading strategy has been designed for the optimal assignment of tasks to machines.Next, Fuzzy logic is implemented to determine the target offloading layers to prevent starvation and aging.The simulation results showcase the efficacy of FedDQL over alternative offloading algorithms based on several performance metrics with different specifications.Tasks and machine heterogeneity are taken into account to appraise the effectiveness of the proposed method.The proposed FedDQL method outperforms others [23], [30], [31], [32], [33] with an average improvement of 49%, 34.3%, 29.2%, 16.2% and 8.21%, respectively.
A potential direction for future study are tasks offloading and data sharing while a vehicle is in motion.Also, data migration between MFC servers, and migration across computing layers in reverse could be a future scope.

is the surrounding noise caused by Algorithm 1 1 . 2 . 3 . 4 . 1 2 5 . 3
FedDQL-Based Offloading Strategy Input: Cluster of Fog zones Z, Participation rate P; Set of vehicles V; Set of requests T; Output: Optimal mapping of subtasks into Fog servers to have reduced energy consumption and computational latency Start Set M = ∅, K = p|Z (f )|; For each Fog zone m ∈ z (f ) Estimate the transmission cost (Tr hp m ) using Eq.(33); Arrange the Fog zones in z(f) according to their transmission cost in ascending order; Discover the initial k number of Fog zones from z(f) and add them into the set of M; 5. for f : 0 to F do 5.For each Fog zone m ∈ M , update its parameter set O f m using the DQN approach 5.Transfer the updated parameter cost to the centralized entity; Aggregation of all the local parameters takes place by the centralized entity as θ f +1 = End other Fog zones.Algorithm 1 shows the pseudocode for the FedDQL-based offloading strategy.