Optimized Processing Placement Over a Vehicular Cloud

Modern vehicles equipped with on-board units (OBU) are playing an essential role in the emerging smart city revolution. The vehicular processing resources, however, are not used to their full potential. The concept of vehicular clouds is proposed to exploit the underutilized vehicular resources to supplement cloud computing services to relieve the burden on centralized cloud data centers and improve quality of service. In this paper we introduce a vehicular cloud architecture supported by fixed edge computing nodes and a central cloud data center. A mixed integer linear programming (MILP) model is developed to optimize the allocation of the processing demands in the distributed architecture while minimizing the overall power consumption. The results show power savings as high as 84% compared to processing in the conventional cloud.Variations in the test cases to include processing demand and traffic demand splitting showed power saving of 71% and 16% respectively, even for large demand volumes. A heuristic algorithm with performance approaching that of the MILP model is developed to validate the MILP model and allocate processing demands in real time.


I. INTRODUCTION
Cloud computing has introduced new possibilities for data processing and storage. This paradigm provides remote services over the Internet that relieve the end users from handling data processing and storage in their own devices [1]. It reduces the cost for users by eliminating the need to deploy and maintain hardware and software resources. On the providers side, the costs of provisioning cloud services are expected to be well compensated through the profit of the growing cloud services. The demand for cloud services is growing exponentially with 30-40% annual traffic growth [2]. Data centers are reported as the main contributors of the total cost and power consumption in the Information and Communication Technology (ICT) field [1]- [4]. In addition, large data centres tend to be located away from the end users, which increases the latency and power consumption of the networks interconnecting users to the cloud. Also, some applications such as smart city produce high volumes of data that have only The associate editor coordinating the review of this manuscript and approving it for publication was Jesus Felez . local relevance making storing or processing them remotely in the cloud an unnecessary burden on the network and cloud resources [3].
Efforts are made to tackle the issues of cloud computing from different perspectives. Mathematical optimization modelling is used as a valuable tool to formulate alternative network solutions with the objective of energy minimization. We benefit from our previous contributions in MILP and energy efficiency for a range of areas. The studies in [5]- [7] looked at green and renewable energy resources in core networks. Analysis of big data and its impact on networks energy consumption is studied in [8]- [11]. Optical networks architecture designs and a number of resilience and fault tolerance schemes are tackled in [12]- [18]. The authors in [4], [19]- [21] designed content distribution schemes for better resources utilization and improved energy efficiency. The authors of [22] modelled energy efficient virtualization in cloud networks. Also, the work in [23] introduced models for virtual machine (VM) placement in cloud-fog networks subject to inter-VM traffic overheads. Paradigms such as IoT [24]- [30] and Fog Computing [31], [32] benefiting from VOLUME 10, 2022 This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/ resources in end users' proximity, are being actively investigated. The resilience of such paradigms is explored in [27], [33] which is an important measure of their reliability and quality of service as alternatives for cloud. The term vehicular cloud [34] was coined to describe the exploitation of underutilized vehicular processing resources to deliver cloud services to end users. This approach usually goes hand in hand with the use of edge and cloud computing [35], [36] to complement the vehicles limited processing capacities, maintain quality of service, and facilitate system management. Our work in [47] introduced an energy efficient vehicular cloud architecture, to be used in distributed processing. In addition to vehicular processing, the architecture provided processing at edge nodes and the conventional cloud. Generic smart city applications were considered to evaluate the performance of the architecture. We evaluated scenarios of multiple requests of varied sizes and showed promising results of 70-90% power savings over the conventional cloud approach for small-sized requests and 20-30% for medium and large requests, respectively. In this paper, more test cases are reported to further establish the merits of the energy efficient vehicular cloud architecture. To the best of our knowledge, work in literature focused on showing the benefits of vehicular clouds but has not provided any solid numerical comparison between the power consumption of the vehicular cloud and conventional cloud, which this paper provides. The contributions of this paper can be summarized as (i) presenting for the first time the mixed integer linear programming (MILP) model developed to optimally allocate processing demands to the three layers of the architecture with the objective of minimizing the power consumption, (ii) comparing the energy efficiency of processing scenarios considering different processing layers, (iii) evaluating different test cases considering varying demand sizes, varying number of demands and the impact of processing demand splitting, (iv) developing a heuristic to allocate processing demands in real time and comparing its performance to the MILP model. The subsequent sections of the paper are organized as follows: The proposed architecture and the MILP model are introduced in Section II. The results of the model are presented and analyzed in Section III. The heuristic and its results are given in IV and the paper is concluded in V.

II. RELATED WORK
Research on vehicular networks has continuously growing. At early stages, the efforts were focused on benefiting from smart vehicle resources in transportation and safety related applications and services [37]. The concept of vehicular networks is still evolving, and research efforts are investigating different perspectives. In [38], [39] the energy efficiency and quality of service (QoS) were studied for different routing and base stations optimization schemes for vehicular networks in a city environment. The same authors in [40] and [41] developed position-based routing schemes for vehicular networks which they proved had a better performance than flood-based protocols. In [42], the authors envisioned the use of resources available in vehicles parked in long term parking of airports and introduced the idea of using these resources as a data center. Also, a two-tier data center system in a parking lot was introduced in [43], in which the cost of communication with the conventional data centers was reduced by using storage resources of vehicles in parking lots instead. In [44], the authors improved the energy efficiency of content distribution to city vehicular users by using renewable energy and adaptive caching points. They also studied the impact of load adaptive caching points on the energy efficiency in [45], and several vehicular network scenarios with energy efficient adaptive/non-adaptive fog servers with renewable/ non-renewable energy in [46]. A multi objective resource allocation model in vehicular clouds was proposed in [48]. The work in [49], [50] explored Sensors as a Service in vehicular networks where the mobility of vehicles broadens the sensors coverage area. Figure 1 presents the proposed energy efficient vehicular cloud architecture and in the following subsections we describe the architecture from three different perspectives: the processing layers, network communication interfaces, and control and coordination.

A. PROCESSING LAYERS
The proposed distributed architecture is composed of three processing layers:

1) VEHICULAR PROCESSING LAYER
The first layer is composed of vehicles equipped with highperformance on-board units (OBUs). Other vehicles can serve the demand if they are willing to share their resources. The vehicles can dynamically cluster to form temporary clouds. Each vehicular cloud is formed under the control of an edge node.

2) EDGE PROCESSING LAYER
The second layer is formed by edge nodes equipped with mini servers dedicated for smart city applications. This layer provides users with another nearby processing destination. In addition to the mini servers, the edge node encompasses an access point (AP) to communicate with vehicles and an optical network unit (ONU) to connect to the passive optical access network (PON).

3) CLOUD PROCESSING LAYER
The last layer is the conventional cloud, which is geographically distant but has powerful computing capabilities. The cloud is connected to a core network node through switches and routers.
The MILP formulation introduced in the next section identifies the optimum solutions given the trade-off between having powerful energy efficient servers located at the cloud, accessed by traversing multiple network layers, and using data rate V2V communication can also be supported through the WiFi interface. Edge nodes, as mentioned above, are equipped with an ONU to connect to higher layers through a PON access network. The inter-communication between the devices composing an edge node is through Ethernet of high speed and low energy per bit, so the power consumed is negligible.

C. CONTROL AND COORDINATION
The decisions of where to serve a user demand? how much of this demand is to be served in a specific location? how the data associated with the processing demand is routed to that location? all fall under the umbrella of system control. These decisions need to be optimized to reduce the power consumption. Generally, the main challenge of the vehicular architecture is the dynamicity and variation of the resources. Keeping track of these changes is crucial to making informed decisions. As previously mentioned, each vehicular cloud is controlled by an edge node. Each edge node is assumed to VOLUME 10, 2022 have knowledge of the vehicles under its control and Edge nodes exchange information about their vehicular clouds resource availability. Based on this information each edge node takes decisions on where to process demands coming from its vehicular cloud. In this work, the overhead created by the control and coordinating data is not considered. In such a distributed control architecture the concept of software-defined networks (SDN) comes into play [52]. The centralized SDN controller, which has a global view of the network is responsible for overseeing and coordinating the edge nodes. Whether using the distributed or centralized approach, the need for dynamic response and frequent updates on the architecture remain the same.

IV. MILP MODEL
The performance of the architecture is addressed through solutions to the problem of resource allocation optimization. The architecture is modeled as a graph G(N , L), where N is the set of all the nodes and L is the link between any two nodes. A node n ∈ N can have a demand that can be serviced in d ∈ N where n = d. Each n ∈ N has processing capacity measured in MIPS and there is a maximum data rate for each link between any two nodes.
A MILP model is developed to optimize the placement of processing demands and route the traffic between the source nodes and destination nodes while minimizing the power consumption, which is composed of processing power consumption and networking power consumption. To the best of our knowledge, this is the first MILP modelled to study the problem of distributed processing in a Vehicular Network environment. The parameters and variables are shown in the next column. Maximum number of processing nodes to process a demand.

B nm
Maximum data rate on the link between n and m (Mbps),∀n ∈ N , m ∈ Nm n . B n Maximum data rate node n can support,∀n ∈ N .

B VE
Maximum data rate of the WiFi interface of a vehicle (this parameter is defined in addition to B n , ∀n ∈ ND to account for the two communication interfaces of vehicles).

B ONU
Data rate of ONU at an edge node (this parameter is defined in addition to B n , ∀n ∈ ED to account for the two communication interfaces of edge nodes).

D nm
Distance between node pair n, m ∈ N . NM n Maximum power consumption of networking at node n ∈ N (W).

NI n
Idle power consumption of networking at node n ∈ N (W).

PM n
Maximum power consumption of processing at node n ∈ N (W).

PI n
Idle power consumption of processing at node n ∈ N (W). NM ONU Maximum power consumption of ONU in edge node.

NI ONU
Idle power consumption of ONU in edge node.

TX
The maximum transmission power consumption of wireless interface.

T nm
Wireless transmission energy per bit over link (n, m) where n, m ∈ ND ∪ ED.

RX
Receiver sensitivity of wireless interface.

R mn
Wireless reception energy per bit at node n over link (m, n) where n, m ∈ ND ∪ ED. Power amplifier factor for wireless communication. E n Energy per bit of networking at node n, n ∈ OLT ∪ MD ∪ CD ∪ ED. PUE n Power usage effectiveness of node n, ∀n ∈ N . A, M large constants.

TP
Total power consumption of the architecture.

W n
Total power consumption at node n ∈ N . WN n Networking power consumption at node n ∈ N . WP n Processing power consumption at node n ∈ N .   β NET n β NET n = 1 if node n is used for networking, n ∈ N , otherwise β NET The work of [76], [77] provided an extensive list of VC smart city applications that range from traffic management, urban surveillance, datacenters, infotainment, healthcare, and emergency management. Similar applications are anticipated to use the services of the proposed architecture. The application demands in this work are generated by the vehicles. However, the model is generic to accommodate situations where the demands are produced by any node type, and this would provide further test cases for evaluation in future work. The demands cannot be locally processed, i.e., it is assumed that a vehicle cannot process its own demands (partly due to capacity constraints and partly to encourage cooperation with other vehicles). However, a vehicle having a demand can still process the demands of other vehicles. Our assumption is that for vehicular clouds to be applicable, cooperation of vehicles owners is required. This can be achieved if they are provided with the proper incentive to provide services to others while discouraging local service of their own jobs.
The demands are assumed to be periodic and subjected to service level agreement SLA specifying the data rate and processing speed requirements. Examples of such demand in public service applications such as CCTV camera systems and climate sensors, to name a few. Accordingly, they are composed of two parts, the data to be sent (in bps) and the processing it requires in MIPS. The two parts are related based on the estimation in [57] for smart environment applications, which are summarized in Table 1. On average, one Mbps is sent for each 2000 MIPS of processing demand. Also, we choose the minimum size of traffic demand as 2 Mbps, which gives a processing demand of 4000 MIPS. The choice is made to allow for distributed processing among vehicles.

B. POWER CONSUMPTION
The processing and networking devices are assumed to follow a linear profile where the power consumption is composed of an idle power consumption which is the power consumed to activate the device and load dependent power consumption as seen in Figure 2. The load dependent power consumption is obtained by multiplying the device load by the energy per bit. For the processing power consumption, the load dependent part is calculated by multiplying the processing demand by processing efficiency. For the wireless communication (DSRC/WiFi), the networking power consumption is also distance dependent as power amplification is required to avoid signal fading over distance.

1) IDLE POWER
The idle power consumption is significant in conventional cloud servers. It is expected to have major impact on the power consumption of the system. Whenever possible, a value for the maximum and idle power consumption is obtained from datasheets when running the model. The idle power consumption value specifically is not always presented. Therefore, idle power consumption for networking devices is taken as 90% of the maximum power, based on estimations in [71]. According to [70], machine to machine (M2M) traffic will be 7% of the global traffic by 2022. Connected cars traffic and connected cities, as part of the M2M traffic, are the fastest growing types of applications. Together they are assumed to make 13% of the traffic [70]. So, for the portion of the idle power consumption of network devices attributed to our application types, we are assuming (0.07 X 0.13) of the total idle power of each device. According to [68], the idle processing power consumption of servers is about 60% of the maximum, but we are assuming a more efficient modern server which consumes an idle power of around 50% of the maximum.

2) POWER USAGE EFFECTIVENESS (PUE)
The PUE is the ratio of the total power consumed by the node's IT equipment (networking and processing devices) and non-IT equipment (cooling, ventilation,..etc) to the power consumed by the IT equipment alone. It is an important measure of efficiency as modern computing and networking nodes require non-computing components for their operation, such as cooling and ventilation systems. An ideal PUE is equal to 1, which means all the power is consumed in performing IT operations. VOLUME 10, 2022 In the following we give a detailed model of the network power consumption and processing power consumption of the different nodes in the network.

3) FOR VEHICULAR NODES
Equation (1) gives the networking power consumption of a vehicular node as the sum of the OBU communication interface idle power consumption, the traffic-dependent, distance-dependent transmission power consumption, and the traffic-dependent reception power.

4) FOR EDGE NODES
The edge node has two communication interfaces, the WiFi interface through its AP, and PON interface through the ONU. In equation (2), the first three terms calculate the AP (WiFi interface) power consumption, while the last two terms are for the PON interface power consumption. The PON interface power consumption is found by multiplying the traffic routed from the edge node ONU to the OLT by the energy per bit of the ONU. The idle power of the ONU is also added to the calculation.

5) WIRELESS TRANSMISSION AND RECEPTION ENERGY PER BIT
The traffic-dependent distance-dependent energy per bit for wireless transmission (T nm ) is given as The first term of equation (3) gives the traffic dependent part found by dividing the transmitter maximum power consumption (TX ) by the link maximum data rate. The second term gives the distance-dependent power consumption as a function of transmission distance and the power amplifier factor.
The far geographical location of the conventional cloud is one of the motivations to have VC and edge processing. Therefore, the physical distance between any two nodes is a parameter that impacts the performance The distance factor has a special importance in VC architectures, due to the ad hoc topology of the vehicles (affecting the networking power consumption), or/and the mobility of the vehicles which makes the distance a changing parameter.
The reception energy per bit for wireless transmission (R mn ) is found by dividing the node receiver sensitivity RX by the link maximum data rate.
The networking power consumption for the nodes from OLT to the core node is calculated by multiplying the networking energy per bit of each node, which is calculated in equations (5-6) by the traffic traversing it, as shown in equation (7).

8) PROCESSING POWER CONSUMPTION
The processing power consumption at any node is given in equation (9) considering the processing idle power consumption and the processing load dependent power consumption which is a function of the node processing efficiency (the power consumed per MIPS as shown in equation (8)).
The objective of the model is to minimize the architecture power consumption as follows: Minimize: where: Equation (10) gives the total power consumption of each node in the architecture. The power consumption is composed of processing-induced part and networking-induced part. In addition, the impact of the power usage effectiveness (PUE) is accounted for at each node, as shown in equation (11). Subject to the following constraints: Constraint (12) states that the processing demand for a source node must be fully served by the processing destinations and the demand cannot be locally processed.
Constraint (13) ensures that the processing demands served by a processing node do not exceed its processing capacity. A binary variable is set to indicate that a node is selected as a processing destination for a demand source as shown in constraints (14) and (15).
The processing demand can be served in several nodes. Each one would receive the full traffic demand from the source, as stated by constraint (16).
Constraint (17) is a flow conservation constraint. It ensures that the amount of traffic received by an intermediate node is equal to the amount re-transmitted. It also ensures that the traffic enters and leaves the node fully at the source and destination nodes respectively.
Constraint (18) ensures that the maximum data rates of the OLT, metro, and core nodes is not exceeded. For vehicles, constraint (19) preserves the DSRC interface data rate (ie ensures that the interface data rate is not exceeded), which is used to communicate with vehicles. Similar constraints for the WiFi interface between vehicles and edge node are separately implemented in constraint (20). Similarly, for the edge node when using the optical communication link through the ONU, the data rate is preserved through constraint (21).
The VC architecture derives its usefulness from the utilisation of number of resources that can collectively deliver services like the ones provided by conventional cloud. To benefit from the distributed processing resources, the division of the processing demand into smaller sub-tasks is allowed. Constraints (22) and (23) Equations (24) and (25) set a binary variable to 1 for nodes used in networking, i.e. transmit, receive, or relay nodes.
Another binary variable is set to 1 in (26) and (27)   A binary variable is set to 1 in (28) and (29) to indicate the use of the ONU in an edge.

V. EVALUATION AND RESULTS
We evaluate the energy efficiency of the proposed architecture considering stationary vehicles in a parking lot. Vehicles in a parking lot offer resources for variable lengths of time, from short-term (half-hour to 3 hours) to long-term (over 3 hours to days and weeks) [53]. This accounts for and is due to vehicular mobility in and out of the car park. These resources are idle in congested business districts (e.g., employees' cars during working hours), or in urban areas with supermarkets and shopping malls, creating opportunities to exploit these resources in smart city applications. Figure 3 illustrates a small parking area of 45 meters X 45 meters, accommodating up to 25 vehicles, with a standard parking space of 4.8 meters X 2.4 meters per car [54]. In our setting, we have 16 vehicles in the parking lot with the distance between two vehicles ranging from 2 meters to 24 meters. The parking lot is surrounded by 4 edge nodes, placed at an average distance of 30 meters away from the vehicles. Both DSRC and WiFi have communication ranges of several hundred meters [55], [56]. Each edge node serves a vehicular cloud of 4 vehicles.
To calculate parameters for the vehicles as shown in Table 2, the following points were considered: begin • Based on [58], highly efficient intel processors execute 4 instructions per cycle. Accordingly, 2 instructions per cycle per core are assumed for OBU in vehicles.
• From [59], OBU processor has 2 cores with Speed = 800 MHz which will be used to calculate the processing capacity in MIPS. Also, Maximum power is (OBUMAX = 10 W) and the idle power is (OBUI = 5 W).
• Based on [46], [60], a general-purpose computer spends 58% of its operational power on processing, 21% on storage (RAM and Disk), 21% on communication. • For the vehicles PUE, the OBU is assumed to be small and efficient enough not to require ventilation system of any significant power consumption. To calculate parameters for the edge nodes as shown in Table 3, the following points were considered: • As stated before, the edge node is composed of access point, server, and ONU, collocated in one place.
• For a server, a raspberry Pi processor is used. • Based on [58], we assume 2 instructions per cycle for the raspberry Pi processor, with 4 cores of speed = 1200 MHz [65], [66].
• The power consumption of the raspberry Pi is dedicated for processing, and this assumption is used to find the processing efficiency. • The three devices of the edge node are collocated in one place, but they are not contained or boxed, which provides natural cooling and ventilation and the PUE can be set to 1.
• The devices of the edge node are integrated together, so the inter-communication between them is ignored in this work.  To calculate parameters for the cloud as shown in Table 4, the following points were considered: • Based on [58], the server is assumed to run 4 instructions per cycle (highly efficient) • The power consumption of the cloud is dedicated for processing, and this assumption is used to find the processing efficiency.
• Transmission power of cloud is ignored as it only receives data, the return of the result (small) to the source node is not considered.
To calculate parameters for metro and core routers and switches, as shown in Table 5, the following points were considered: • The values for the PUE in the network devices (routers and switches) are derived from [4] • For switches, it is assumed the devices become more power hungry as they get closer to the core and datacentre. For example, core switches are in the backbone of the network and deal with multiple aggregations and require more complicated functionalities for security, fault tolerance, and networking. Therefore, the aggregation switches are set to consume the typical power values in [72] and the cloud switches consume the typical power value stated in the datasheet [72].
• For the core and cloud routers, the port power consumption is estimated from [73], by diving the total power consumption (1450 W) by the number of ports (48 ports) as all ports have the same capacity.
• For the aggregation router, the port power consumption is estimated from [73], by dividing the maximum power consumption (420 W) by the maximum throughput (800 Gbps) to get the W/Gbps, which was then multiplied by the port capacity of 10 Gbps.
• The processing capacity of OLT, routers and switches in metro and core is set to zero. The main aim of this work is to produce quantitative results that determine the power consumption in the distributed resources in the VC and edge, in comparison to the use of the conventional cloud as baseline. The evaluation test cases were chosen to reflect several factors that were expected to have an effect in a distributed architecture such as traffic demand size, processing demand splitting and the competition over resources between multiple demands. We have test cases with only one vehicle generating a demand and other cases where we have demands generated by several vehicles. The number of splits allowed in the processing demand provide another evaluation perspective. As the processing demand is split, the associated traffic is delivered to each allocated destination. Also, the traffic associated with processing demand can be delivered fully or partially to the allocated processing node. An example of the full traffic delivery is a pedestrian collision avoidance application where one image is delivered to two processors and one of the processors is assigned to search for pedestrians for example, while the other processor searches for vehicles. An example of partial traffic delivery is sending half of the image to one processor and sending the other half of the image to the second processor where each processor searches for pedestrians in front of vehicles.
We evaluate 4 scenarios of processing resources availability. In the first scenario requests can only be processed in vehicles (V scenario). The second scenario optimizes the allocation of the processing request at vehicles and edge nodes only (VE scenario). In the third scenario, only conventional cloud has processing capacity (C scenario). The C scenario is the one used as benchmark for comparison. The last scenario optimizes the allocation of processing resources at the three processing layers (VEC scenario). The following sections present the results obtained from running the model with the above-mentioned scenarios over number of test cases.

A. DEMAND SIZE VARIATION
We consider a single vehicle to generate the demand. The processing demand is varied between 4000-60000 MIPS to reflect tasks with low processing requirements to high processing requirements. A processing demand can be split among any number of processing destinations, i.e. S is set to the maximum number of available processing nodes in the MILP model.
For the V scenario, the results showed that the model saturates the vehicles in the same vehicular cloud of the vehicle generating the demand before moving to other VC. In the VE and VEC scenarios, edge nodes are preferred because of their higher processing capacity and efficiency, so one edge node is packed before activating more vehicular processing nodes as the demand grows larger in size. Figure 4 shows the total power consumption, with processing is the dominating contributor to the power consumption. The figure shows the merits of having the edge nodes as supporting processing resources that further lower power consumption due to the tendency to consolidate the demand to avoid more node activation and increase in idle power. The VE scenario matches the optimal solution given by the VEC scenario, except for the last demand where the optimal solution is to serve the request in the cloud due to capacity limits of the vehicle and edge nodes.
Further inspection of the figure shows that for the V scenario, the total power exceeds the power consumed in the conventional cloud scenario. Also, it is worth mentioning that for the larger demands not served in the V and VE scenarios, the bottleneck is the networking capacity and not the processing capacity, e.g. for the largest traffic demand of 30 Mbps, the associated processing demand is 60000 MIPS while the total capacity of vehicles and edge nodes is 86400 MIPS. Therefore, increasing the bandwidth capacity can go a long way in improving the performance and power saving of the architecture.
In Figure 5, we break down the networking power consumption. Figure 6 shows a surge in the networking power consumption of the V scenario serving a traffic demand of 8 Mbps. This is because 5 processing vehicles are  needed to fully serve the demand (8 Mbps is associated with 16000 MIPS). This generates a total traffic of 40 Mbps to be transmitted from source to the processing destinations, which exceeds the capacity of the DSRC and necessitate the use of the WiFi and therefore leads to the increase seen in the power consumption. The other surge of the V scenario at 14 Mbps is due to edge nodes communicating through the PON access networks as the edge nodes WiFi APs cannot support this data rate. Figure 6 breaks down the processing-induced power consumption. For the VEC scenario, the vehicle OBU are optimal as long as the total idle power of processing vehicles is lower than that of a single edge node. For example, at 2 Mbps (4000 MIPS), it is optimal to split the demand between two vehicles, while for the 4 Mbps demand (8000 MIPS), activating one edge server is more efficient than activating 3 vehicles OBUs. A combination of OBUs and edge node servers resulting in minimum power consumption is activated to serve higher processing demands. Figure 7 shows the power savings of the three distributed processing scenarios (V, VE and VEC) in comparison with processing in the conventional cloud. Optimized processing  in the VEC scenario resulted in power savings up to 84% compared to processing in the cloud. Limiting processing to the vehicles and edge nodes (VE scenario) gave the maximum power savings for traffic demands as high as 20 Mbps. The energy efficiency of the vehicular only processing decreases as the size of the demand increases. Processing a demand of 14 Mbps in the vehicular cloud proves to be less efficient than cloud processing by 23%.

B. PROCESSING DEMAND SPLITTING LIMITATION
Having the flexibility of splitting processing demands can improve resource utilization and consequently the energy efficiency. It can also reduce the total processing time and avoid exhaustion of computational resources. In this subsection we study the impact of limiting the number of processing nodes that can serve the request as opposed to the unlimited processing demand splits studied in the previous subsection. Figure 8 shows the total power consumption of the VEC scenario under different splitting limits. It shows that a splitting limit of 2 is enough to achieve the minimum power consumption in this case. Splitting a request of 5.5 Mbps traffic demand between two processing destinations in a VEC scenario improves the energy efficiency by 71% compared to processing without splitting which results in processing the request in the cloud as seen in Figure 9.
From Figure 9, we can also infer that the usability of V scenario is limited when no splitting was allowed. Splitting traffic demands of 2-3 Mbps allows them to be processed in  the vehicular cloud which slightly increases the processing power consumption, as seen in Figure 9, due to the lower processing efficiency of the OBUs. On the other hand, avoiding communicating with the edge nodes by processing in the vehicular cloud significantly reduces the networking power consumption as seen in Figure 10 although traffic is replicated to the two vehicular processing destinations serving the demand. The overall power reduction from the no splits case is 2-3%.

C. PROPORTIONAL TRAFFIC ASSIGNMENT
In all the previous subsections, we assumed that all processing destinations receive the traffic demand in full, even when serving part of the processing demand. This limits the efficiency of processing demands splitting as it burdens the network. However, for some types of applications, a processing node serving part of the processing demand requires access to only part of the data to be processed. An example of this can be an application processing multiple images or multiple videos as discussed earlier. In this subsection we optimize the processing of a demand with traffic that can be proportionally split among the nodes serving the demand, referred to as proportional traffic (PT) demand. This test case is compared to the one where the full traffic (FT) demand is delivered to every processing destination.
The MILP model is updated to represent PT requests, by replacing constrain (16) with the following equation: Constraint (30) ensures that the traffic delivered to a processing node from a source node is proportional to the processing carried by the processing node.
The comparison considers a single demand of varying sizes for the 3 scenarios (V, VE, VEC), the C scenario is unaffected by this change as the whole demand is sent to the cloud. Figure 11 compares the FT and PT cases under the V scenario in terms of processing and networking, respectively. Proportionally splitting the traffic among the processing destinations has not changed the number of processing destinations compared to the FT case. However, the networking power consumption was reduced. It improved the utilization of the DSRC communication bandwidth and therefore reduced the networking power consumption as the WiFi interface and the PON network are used less. Proportionally splitting traffic also relieves the traffic bottleneck observed for FT case for traffic demands higher than 14 Mbps. Similar trends are observed for the VE scenario (not shown in the figures), which improved utilization of the network bandwidth under the PT case and allowed the VE scenario to serve 30 Mbps demands.
Comparing the PT and FT cases under the VEC scenarios in Figure 12 confirms that the cloud is the optimal processing destination for the 30 Mbps demand even when the demand traffic resulting from distributed processing in the vehicular and edge layers can be supported by the network. This is due to the energy efficiency of the cloud in processing such a large demand compared to processing in the vehicular cloud and edge nodes which requires activating multiple processing destinations.
Proportional traffic impacts the power savings achieved by the different processing scenarios compared to processing in the conventional cloud, as seen in Figure 13. Proportionally splitting the traffic improves the energy efficiency of the V scenario compared to the FT case making processing demands as high as 20 Mbps in the vehicular cloud more efficient than cloud processing. As mentioned above, under the PT case the VE scenario can process demands as high as 30 Mbps. This is, however, less efficient than processing in the cloud by 14%. For the VEC scenario, the power savings improved from 6% for the FT case to 19% for the PT case for a demand of 20 Mbps compared to processing in the cloud.

D. MULTIPLE DEMANDS SERVICE
In this section we examine multiple requests competing for the available resources under the VEC scenario considering full traffic replication to all processing destinations. Relative to the processing resources in vehicles and edge nodes, three demand profiles are examined: low (Traffic 1 Mbps, Processing 2000 MIPS), medium (Traffic 3 Mbps, Processing 6000 MIPS), and high (Traffic 5 Mbps, Processing 10000 MIPS).The low demand can be served in single vehicle, medium demand can be served in single edge node, and the high demand exceeds the capacity of a single edge node. Figure 14 evaluates the total power consumption of up to 10 co-existing requests and illustrates the impact of varying demand size/number on the processing and networking power consumption. With the growing requirements, it becomes less costly to operate the cloud as the difference between the high idle power consumption of the cloud server and the idle power consumption of the multiple vehicles and edge nodes required to serve the demands decreases. Also, for higher demand sizes the increase in the networking requirements cannot be accommodated with the use of the vehicles and edge only. The power savings of the VEC scenario rapidly decreases as the demands increase in size/number, as shown in Figure 15. For medium and high demands as the increase in the number  of demands forced the use of the cloud, the power savings drops to 0%.

VI. ENERGY EFFICIENT DEMAND ALLOCATION HEURISTIC IN A VEHICULAR CLOUD ARCHITECTURE
A heuristic is developed based on insights obtained from the model to allocate processing demands in real time. The heuristic flowchart is shown in Figure 16. The heuristic serves demands in descending order of their processing  requirements as the processing power consumption dominates the power consumed to serve demands. For each demand, the processing nodes are sorted based on the criteria defined in equation (31), with the most fit candidate in the beginning of the list and the least fit at the end. Sorting of the processing nodes can be ascending or descending depending on the demand size in comparison with the capacities of the distributed processing nodes. Also, the counter (Trial) is set to 1. This variable is set to give each demand two attempts to find a processing destination(s) by going over the complete list of processing nodes candidates and trying to route over them.
SortingCriteria sd = NPower sd + PrPower sd +Idle d PUE d ∀s, d ∈ N , s = d (31) where: NPower sd is the power consumption of routing traffic between source s and processing destination d over the minimum hops route.
PrPower sd is the processing power consumption of serving processing demand of source node s in processing destination d.
The heuristic selects the most fit candidate from the sorted list and tries to route the traffic demand over the minimum hop route if the candidate has available processing resources. If the minimum hop is not available due to networking capacity limitations, the heuristic removes the link of limited capacity from valid routes. The heuristics then selects the next node in the sorted list and tries to route the traffic demand on the minimum hop route. The heuristic examines all the nodes in the sorted list until all the demand under consideration is served. If all nodes are examined but the demand is not fully served, the trial counter is incremented by 1 and the heuristic examines nodes in the sorted list again. Note that the availability of the minimum hop routes will change in the second attempt, giving the processing nodes that were skipped before due to networking limitation a new chance in terms of the least hops route, and the processing node might be used to serve. Each demand is allowed only two attempts (more attempts can be allowed, at the cost of increased complexity) of examining the nodes in the ordered list to select a processing destination(s). Also, the number of processing nodes used is counted to ensure they do not exceed the allowed split limitation. Furthermore, a processing node is packed before moving to the next node.
If the demand is not served after the two attempts, it will be blocked and all the resources that were used to partially serve it are released to be used by other demands. The heuristic then moves to the next demand on the demands list and repeats the above procedure. After completing all the demands list, the total power consumption resulting from routing all demands is calculated. Figure 17 shows a comparison between the total power consumption of the heuristic and the MILP model results when considering a single request of varying size in the VEC scenario. Table 6 shows the gap in power consumption between the two. The heuristic approaches the model with a gap of 0%-15% for most of the demand sizes. This gap is a result of the heuristic sub-optimal selection of processing nodes, as seen in Figure 18, resulting from the sequential allocation of the processing based on the current status with no knowledge of the upcoming demands. For the heuristics there is a pattern of depleting the resources of the lower processing layers before allocating demands to higher processing layers. On the other hand, the MILP model allocates demands to edge nodes or a combination of vehicular and edge nodes even when vehicular nodes have more available resources.

A. DEMAND SIZE VARIATION
The 30 Mbps demand is optimally processed in the cloud, both in the model and the Heuristics. In this case, the traffic demand exceeded the capacity available in the DSRC, which led to the list of candidates to be sorted in descending order, as stated in the flowchart. This arrangement pushed the cloud node to the front of the sorted list, and it was chosen as destination to serve the demand. If the processing nodes were sorted ascendingly, the cloud would have been at the end of the candidate list. As the heuristic would distribute the  demand between the vehicles and edge node and when it tries to serve the remainder of the demand in the cloud, it finds the network capacity at the vehicles and edge node layers was already occupied by the traffic of distributed processing in the vehicular and edge layers.

B. PROCESSING DEMAND SPLITTING LIMITATION
As part of taking stock of the available resources, the heuristics counts the minimum number of nodes needed to process a specific demand in each processing layer. When it checks the candidate processing capacity, it also checks if the number of nodes required at the candidate layer is within the splitting limitation. If not, then it is seen as insufficient, and the heuristic moves to the next candidate. Table 7 shows the gap between the power consumption values. For smaller demand values, the results are identical for all splits limitations. For demands of 3.5 Mbps and above, the results were identical for smaller number of splits, but as the number of splits increased, the heuristics produced higher power consumption. The reason for this is that as the limits on the number of processing nodes becomes larger, the possibility of serving in the vehicles increases. The heuristics only ensures that the number of processing nodes does not exceed the limit, while the MILP tends to consolidate the demand in a destination whenever possible, leading to fewer number of active nodes and lower power consumption.

C. PROPORTIONAL TRAFFIC ASSIGNMENT
To implement the heuristics in this case, the same change that was made in the model in equation (16) was also made VOLUME 10, 2022   in the heuristics so that each destination receives traffic proportional to the processing demand it serves. The results of the heuristics for the proportional traffic in the V scenario showed a complete match with the results of the MILP model. The results for the VE scenario showeds higher power consumption in the heuristic, with the difference shown in Table 8. As was explained before, the difference in the power consumption comes from the sequential allocation of the demands in the heuristics. Even though it is not shown, the VEC scenario had an exact match with the results of the V and VE scenarios in the heuristics, except for the 30 Mbps demand size, for which it made the same choice as the MILP and processed in the cloud.

D. MULTIPLE DEMANDS SERIVCE
Results for the multiple demands were obtained from heuristics using VEC scenario. As before, the heuristics sequential approach gave priority for processing in the vehicles before moving to the edge and then the cloud. The distribution over more processing nodes and the choice of first sufficient route with least hops increases the power consumption in the heuristics. The behaviour is similar for low, medium, and high demand sizes. However, for the high demand sizes with the number of requests above 5, the demand requirements exceeded the capacities of the vehicles and edge layers. The heuristics sorted the candidates in descending order (refer to the flowchart), therefore, the demands for these cases were served in the cloud, similar to the model. The higher power consumption of these cases is due mainly to the routing decisions and not processing. Table 9 shows the difference in the power consumption between the optimal solution (MILP) and the heuristic.

VII. CONCLUSION
This paper has investigated the use of underutilised computing resources in modern vehicles to create a processing layer, referred to as the vehicular cloud, in proximity of end users. The vehicular cloud complements conventional cloud computing and fixed edge computing in a distributed processing architecture. The architecture was modelled using a MILP model with the objective of minimizing the total power consumption. The results of the MILP model show that the energy efficiency of processing in vehicles compared to the cloud decreases as the size of the demand increases. Processing in a combination of vehicles and edge nodes results in average power savings of 6% compared to processing in the cloud for demands of traffic as high as 18 Mbps. The limited data rate of the vehicle wireless interfaces cannot support distributed processing in vehicles and edge nodes as the traffic is replicated to all processing destinations. Therefore, vehicular communication interfaces of higher data rate are essential to improve the utilisation of vehicular clouds. The results also illustrate that splitting a processing demand improves the energy efficiency of processing in the vehicles and edge nodes by 71%. Furthermore, the results show applications which require proportional traffic splitting among the processing destinations serving the demand. These applications can be more efficiently processed by vehicles and edge nodes, thus increasing the average power savings to 3%-16% compared to cloud processing, even for demands up to 20 Mbps. A realtime heuristic for allocating processing demands is developed based on insights from the model. The results show that the heuristic has comparable performance to the MILP model.