Towards a Green Approach for Minimizing Carbon Emissions in Fog-Cloud Architecture

Fog computing is developed to complement cloud computing by extending the cloud services (computing, storage, networking, and management) to the edge of the network in order to reduce service latency. Correspondingly, the incremental use of cloud/fog resources and their applications has increased energy consumption and carbon emissions (CO2) of the data centers, which caused significant environmental challenges. Optimizing the placement of the requested resources and applications (e.g., in the form of virtual machines (VMs)) is one of the main solutions, which has a primary effect in reducing the energy consumption of cloud/fog architectures and consequently their CO2 emissions. However, due to the geographic distribution of cloud and fog data centers, there are varying levels of CO2 emissions to consider, which makes optimizing the placement of resources and applications in distributed cloud/fog more challenging than in centralized clouds in terms of carbon efficiency. In this paper, we propose a multi-level approach using a mixed-integer linear programming (MILP) model to minimize the CO2 emissions of data centers by optimizing the resources usage and the placement of VMs in fog-cloud environments. This approach calculates the CO2 emissions of the British Telecom (BT) network based on the carbon intensity data from the National Grid ESO, considering several scenarios of traffic demand during different times of the day and year. The results show that the optimal location to host applications highly relies on the carbon intensity and traffic demands. The results also show there is a trade-off between CO2 emission reduced by shortening network journey, and CO2 emission increased by hosting more applications into the fog nodes. In addition, the results demonstrate that the proposed green fog-cloud architecture outperforms the central cloud and the distributed clouds in terms of reducing the total CO2 emission by up to 91% and 71%, respectively. Finally, we develop a heuristic algorithm to mimic and validate the presented work, and it shows comparable results to the MILP model.


I. INTRODUCTION
The advancement of online applications has resulted in massive amounts of traffic generated with significant volume. Therefore, cloud computing is considered one of the most significant information and communication technology (ICT) paradigms, that allows users to share resources and various services including computing, networking, and data storage [1]. However, the long-distance between the end-users and the cloud may lead to a huge latency, which has a negative impact on service performance (e.g., time-sensitive applications) [2]. Consequently, fog computing has emerged to The associate editor coordinating the review of this manuscript and approving it for publication was Guangjie Han . complement cloud computing, which considers the requirements of transferring a huge amount of data as well as the network latency [3]. The main feature of fog computing is to extend cloud services to the edge of the network (closer to end-users) [3], [4]. Thus, proposing fog computing has provided many advantages such as real-time processing, rapid scaling, and resource pooling [5], in addition to increasing the security and density of devices and mobility [6].
As demand for ICT is constantly mounting globally, the request for computing resources globally has increased six times since 2010 [7]. With the increase in demand for online computing resources, the escalating power consumption is becoming a crucial concern for both operators and governments in term of energy costs and carbon emissions (CO 2 ). In response, new approaches for orchestrating data centers resources usage are required to limit their increasing environmental impacts [3], [6].
Virtual machines (VMs) consolidation is considered as one of the most productive methods to enhance resource usage and reduce energy consumption in both cloud and fog environments, as the virtualization technology enables to create, manage, and remove the requested resources flexibly over the physical servers, storage, and networks [1], [3]. However, the increased network traffic generated by cloud and fog services leads to the growth of network power consumption, especially considering the traffic demands between the cloud/fog resources and end-users [1].
Therefore, the optimization of both network and cloud/fog resources is essential, to achieve a green fog-cloud architecture. In this regard, many research works have studied this area. For example, the authors in [1] find the optimum location to place the VMs in the cloud data centers with the aim to minimize power consumption. They developed a mixed-integer linear programming (MILP) model to optimize the placement of VMs within the clouds under various traffic demands throughout the day. They found that the optimal VM placement decision highly depends on the tradeoff between the power saved by the network and power consumed by hosting extra VM copies. Further, the authors in [8] have optimized VM services to reduce the energy consumption of cloud-fog architecture. In their design, different download rates and different VM types have been considered. They discovered that the decision of offloading VMs to fog nodes highly depends on various factors such as the workload of the VM and the proximity of the fog node.
Additionally, improving carbon emissions over cloud and fog data centers has become a significant challenge that attracts the attention of many researchers. Several works have investigated reducing carbon emissions in cloud and fog environments. For instance, the authors in [9] have used renewable energy in fog architecture to reduce power consumption and CO 2 emissions for video streaming applications. They found that under the same power usage effectiveness, the power is saved by 70% in solar-powered fog data centers. Also, the authors in [10] have used solar and wind renewable sources to achieve a green Internet protocol (IP) over wavelength division multiplexing (WDM) networks. A linear programming (LP) model is developed to improve the utilization of non-renewable energy and consequently reduce CO 2 emissions.
In this paper, we propose a multi-level approach using a mixed-integer linear programming (MILP) model to achieve the green fog-cloud architecture. The CO 2 emissions per kilowatt-hour on an electricity grid varies by time of day and significantly fluctuates by location due to the types of generation. Thus, we can exploit that fluctuation by effectively placing the applications (VMs) based on ''where'' and ''when'' the electricity grid is less carbon-intensive, which can contribute to reducing the overall CO 2 emissions of the data centers. In this work, we consider different CO 2 emissions per kilowatt-hour of different United Kingdom (UK) cities, which is obtained from National Grid ESO [11]. Based on these data, we optimize the placement of applications over cloud and fog environments, considering different applications requirements with different data rates. The main contributions of this paper are summarized as follows: • Design an efficient architecture based on mathematical modeling and heuristic algorithm to investigate the placement of cloud/fog applications, considering the minimization of the overall CO 2 emissions of networking in different cities.
• Optimize the placement of applications over the proposed fog-cloud architecture in order to reduce the associated CO 2 emissions, which would also help the cloud/fog providers to find the optimal geo-locations to place their applications and reduce costs.
• Evaluate the effectiveness of the proposed approach in terms of reducing CO 2 emissions, and compare the obtained results with the central and distributed cloud architectures. The remainder of this paper is organized as follows: Section II introduces the green approach for fog-cloud system architecture, considering the British telecom topology. Section III presents the MILP model for optimizing the placement of the applications in the fog-cloud architecture. Section IV shows the model design and its input parameters. This is followed by a discussion of the optimization model results and analysis in Section V. In Section VI, we introduce the heuristic of the green application placement over the fog-cloud architecture (HOGAP-FC). Finally, Section VII concludes the paper and discusses future work.

II. A GREEN APPROACH FOR FOG-CLOUD ARCHITECTURE
The green fog-cloud architecture is illustrated in Fig.1, which includes three levels: the users' level, the fog level, and the FIGURE 1. A green fog-cloud architecture. VOLUME 9, 2021 cloud level. The description of each level of the proposed architecture and their roles are presented as follow: A. USERS' LEVEL The first level that is described in the proposed architecture is the users' level. The users' level is the lowest level and has the job of requesting applications in the form of VMs, and these VMs will be placed afterward into suitable fog or cloud nodes for processing.
It is worth mentioning that one of the main benefits of virtualization is to abstract the server resources (like the processor, memory, disk, and network) in order to create isolated VMs that can run their own operating systems, services, and applications requested by users. The existence of such a virtual environment allows the placement of the VMs over geo-distributed servers (cloud and fog) to achieve different aims such as load balancing [12] and energy efficiency [13]. However, for VMs that require real-time processing (e.g., time-sensitive applications), service placement is very crucial [14].
In fact, the VMs placement is a challenging task in both the cloud and fog environments [15]; especially, in terms of reducing the energy consumption and CO 2 emissions of their data centers [3]. Thus, optimizing VM placement over cloud and fog architectures can significantly reduce energy consumption and service latency; consequently, reduce the carbon emissions and operating costs of the data centers.
In this design, the placement of applications is generally carried out using VMs by orchestrating the usage of cloud and fog nodes, considering certain constraints (e.g., CO 2 emissions and data rates). Also, a decision on the placement of the VMs is taken in real-time in order to place the VMs on the servers (cloud and fog) located in cities that have fewer carbon emissions.

B. FOG LEVEL
The word ''fog computing'' was firstly used by Cisco in 2014 [16]. Fog computing expands the cloud computing resources to the edge of the telecommunications network to improve the efficiency of the services and lower the latency of the applications [17], [18]. Both fog and cloud computing can provide storage, computing, and networking services to the users' level. In addition, fog can be deployed in several locations (cities).
The concept of fog computing means that, instead of executing all the applications (VMs) in a centralized cloud, fog can be more feasible to process the VMs locally and close to the users to reduce bandwidth issues. Also, due to the geographic distribution of fog computing, different areas have different carbon emission rates and energy costs that need to be considered, which makes optimizing the placement of VMs in distributed fogs more challenging than in centralized clouds in terms of carbon efficiency [3]. In this context, fog level provides location awareness due to the localization of fog nodes in different cities (geographical distribution), considering both the CO 2 emissions and network traffic.

C. CLOUD LEVEL
At the same level of importance as fog, cloud computing essentially enables the growth and development of VMs applications. It provides resource-sharing and elastic services (e.g., network, storage, and computational) to many geographically distributed users. However, the main challenge of using cloud computing services on a large scale is the issue of networking bandwidth (service latency) for VMs applications [15].
The total application traffic is estimated to be raised to 3.7 times in 2022 compared to its level in 2015, reaching 400 Exabytes per month [19]. This increasing traffic creates a huge burden on data centers and networks, leading to serious energy consumption and CO 2 emissions challenges [20], [21].
Thus, cloud services need to improve their network issues, where fog computing can be beneficial to address these challenges. For example, in this work, the applications can be placed in the cloud or pass the requests to the nearest available fog nodes within the user's area, which will be based on the lowest CO 2 emissions of that area. Also, this will help to accelerate and improve response time and decision-making by processing the applications within the local network, consequently reducing network traffic.

D. TELECOMMUNICATION NETWORKS
Based on the traditional telecommunication network architecture presented in Fig. 1, the telecom network consists of three layers: core network (the wide-area network (WAN)), metro network (the metro area network (MAN)), and access network (the local area network (LAN)). The core network (WAN) is considered the backbone infrastructure of the whole telecom network since it interconnects major cities. Also, the IP over the WDM, which is the main technology used in the WAN network provides wide capacity, high scalability, and fast communication to support the transfer speeds of the WAN network [4]. Each WAN network connects with a MAN, which provides a direct connection between the WAN network and its residential users (users located in the LAN network area). The access network (LAN) supports Internet access to numerous user premises, in which passive optical network (PON) technology is mainly used. The PON includes two parts: optical line terminal (OLT) and optical network unit (ONU), which are close to end-users.
In the proposed fog-cloud architecture, cloud data centers are deployed in the WAN networks and fog data centers are deployed in the MAN networks in order to extend the clouds to the edge of the network.

E. BRITISH TELECOM NETWORK TOPOLOGY
The British Telecom (BT) network topology, which is introduced in Fig. 2, consists of 20 nodes (cities) and 68 bidirectional links [22]. In this work, we aim to achieve a green fog-cloud architecture by optimizing the placement of cloud/fog applications over the BT network, considering several factors: data rates, carbon intensity, and the proximity of data center locations. The total number of users in the network fluctuates based on the time periods they access the applications. For instance, 15 million users at 10 AM and 45 million users at 10 PM. We consider two data rates for users to access the network: 0.5Mbps (low) and 5Mbps (high), representing light web browsing and video streaming, respectively. The dividing of users accessing the applications is 70% for light browsing applications and 30% for the video streaming applications. Although the carbon intensity is fluctuant from day to day, we consider the carbon intensity of selected days of 2020: 1 st of Jan, 1 st of Apr, 1 st of Jul and 1 st of Oct. These days represent winter days, spring days, summer days, and fall days, respectively, through the year.

III. MILP MODEL FOR OPTIMIZING APPLICATIONS PLACEMENT BASED ON CO 2 EMISSIONS OVER FOG-CLOUD ARCHITECTURE
In this section, a multi-level approach is developed using a MILP optimization model to investigate the placement of applications over the fog-cloud architecture, considering the three telecom networks: LAN at the users' level, MAN at the fog level, and the WAN at the cloud level.
In the following, we present the parameters and variables of the proposed architecture, which include the users, fog, and cloud levels. Then, we provide the mathematical model to find the optimal placement of applications based on their data rates and the CO 2 emissions of different cities over the fog-cloud architecture.

A. USERS, FOG, AND CLOUD LEVELS
The following parameters and variables (in Tables 1 and 2) represent the applications that will be placed in the fog and cloud levels, as well as the resulting traffic and power consumption.

1) CARBON EMISSIONS (CO 2 ) OF FOG-CLOUD LEVELS
Carbon emission [23] is defined as the CO 2 emissions intensity for the amount of energy consumed. The CO 2 emissions intensity is measured by kgCO 2 e/kWh. In the following Table 3, we define the parameter related to carbon emissions in both fog and cloud levels.
The carbon emission formula of fog and cloud nodes consist of: Cloud level (cloud): (2) Equations 1 and 2 calculate cloud and fog computing emissions, respectively, taking into consideration both processing, networking devices, as well as PUE of cloud and fog levels.

B. TELECOMMUNICATION NETWORKS
As described in Section II-D, a typical telecom network is considered, including WAN, MAN, and LAN networks. The traffic traversing through these layers as well as the corresponding power consumption are represented by the parameters and variables described below.

1) LOCAL AREA NETWORK (LAN)
The parameters and variables that define the LAN network are shown in Tables 4 and 5.
Local area networks carbon emissions (LAN) consists of: Equation (3) calculates the total carbon emissions of the LAN network, including Optical Network Units (ONU) and  Optical Line Terminals (OLTs) devices, taking into consideration the network PUE.

2) METRO AREA NETWORK (MAN)
The parameters and variables introduced to define the MAN are shown in Tables 6 and 7.  The metro area network (MAN) carbon emission formula is: Equation (4) calculates the total carbon emissions of the metro network, taking into consideration metro edge router and switch devices as well as the network PUE.

3) WIDE AREA NETWORK (WAN)
The parameters and variables introduced to define WAN networks are shown in Tables 8 and 9.
The wide area network (WAN ) [24] carbon emission formula:  Equation (5) calculates the total carbon emissions of the core network, including core router ports, transponders, amplifiers, and switch devices, taking into consideration the network PUE.
The MILP model, considering the equations from (1-5), is represented by the following: The objective: Minimize total CO 2 emissions: Expression (6) calculates the carbon emission of our proposed architecture as the sum of the carbon emissions of the core network, the metro network, the LAN network, cloud, and fog levels.
Subject to the following constraints: Traffic in cloud/fog architecture: Constraint (7) guarantees that all the users processing requirement is processed in a cloud or fog node.
Application placement in cloud/fog architecture: Constraints (8) and (9) Constraints (10) and (11) ensure that the physical link (m, n ∈ cloud) is activated if there is a traffic flow between the nodes (s, d ∈ cloud) transmitting through the physical links (m, n ∈ cloud).
Fog/cloud processing requirements: Constraints (12) gives the processing requirements of the application i ∈ app in the cloud and the fog levels. Constraint (13) gives the total processing requirements of a cloud and a fog d ∈ N .
Traffic demand on WAN network: Constraint (14) calculates the traffic demand between WAN nodes due to the applications placed in the clouds. Constraint (15) defines the flow conservation of the core network. It ensures that the total incoming/outcoming traffic in all core nodes is identical, excluding the source/sink nodes.
Physical link capacity: Constraints (16) gives the physical link capacity by ensuring that the traffic in a link does not exceed the maximum capacity of fibers.
Total number of router ports in a WAN network node: Constraint (17) gives the router port count in each core node.
Total number of ONU terminals in each node: Constraint (18) gives the number of used ONU terminals in each node. VOLUME 9, 2021 Total number of OLT in each node: Constraint (19) gives the number of used OLT in each node. MAN routers: (20) Constraint (20) gives the count of metro edge routers in each node.
MAN switches: Constraint (21) gives the count of metro ethernet switches in each node.

IV. MILP MODEL DESIGN
This section describes the design of the experiment and the details of the models conducted in order to evaluate the green fog-cloud architecture.

4) DESIGN OF THE EXPERIMENT
In terms of the experiment environment, the MILP model is implemented using the CPLEX solver over a PC with (Intel Core i7-7660U CPU, 2.50 GHz, 16GB RAM, and 1TB SSD).
All CO 2 emission data used in this work is from National Grid ESO [11]. We considered the carbon intensity of 20 cities around the UK to represent the core nodes of the BT network. The data have been taken at 10 AM (morningtime) and 10 PM (night-time), on 1 st of Jan, 1 st of Apr, 1 st of Jul, and 1 st of Oct, during winter, spring, summer, and fall seasons, respectively.
In this design, we considered two types of network traffic: high (5 Mbps) and low (0.5 Mbps). The total number of users for all the applications (VMs) is 60 million. Moreover, the number of users is divided into 2 groups: 30% of users used high network traffic (5 Mbps), where 4.5 million users accessed the network at 10 AM and 13.5 million users accessed the network at 10 PM. In addition, 70% of users used low network traffic (0.5 Mbps), whereas 10.5 million users accessed the network at 10 AM and 31.5 million users accessed the network at 10 PM.

5) INPUT PARAMETERS OF THE MODELS
In the MILP models, we have configured three levels in the green fog-cloud architecture, which is composed of the users' level, the fog level, and the cloud level, as shown in Section III. In this subsection, the configuration of networks (LAN, MAN, and WAN) input parameters are defined in Tables 10, 11, and 12, respectively. Additionally, cloud and fog input parameters are shown in Table 13.

V. RESULTS AND DISCUSSION
In this section, we discuss the proposed fog-cloud architecture in terms of its CO 2 emission efficiency and evaluate  our proposed architecture to optimize the placement of the applications compared to the central cloud and the distributed clouds.

A. OPTIMAL APPLICATIONS PLACEMENT BASED ON CO 2 EMISSION
This subsection demonstrates the optimal applications placement over the fog and the cloud, considering different traffic profiles (high and low data rates), at 10 AM and 10 PM, during 1 st of Jan, 1 st of Apr, 1 st of Jul and 1 st of Oct in 2020, which represents the seasons of winter, spring, summer, and fall. Fig. 3 shows the optimal applications placement over fogcloud architecture based on the emission values at 10 AM 1 st of January 2020 stated in [11], taking into consideration 0.5 Mbps and 5 Mbps traffic rates. In the figure, the x-axis is the set of cities on the BT telecom networks, the primary y-axis shows whether cloud and/or fog is built in a certain location, the secondary y-axis is the CO 2 emissions in each city, which is obtained from [11].
Under high traffic rate, Fig. 3 shows that 2 clouds and 9 fog nodes are selected to host applications. Cities with emissions less than or equal to 150 Kg CO 2 e/kWh are served from a local fog node (9 fog nodes in total), as the emissions values have justified the creation of multiple fog nodes. In clouds, Glasgow and City Valley are selected to host application copies as they have the lowest carbon intensity (24 Kg CO 2 e/kWh). These clouds' locations are created to serve cities that have emissions higher than 150 Kg CO 2 e/kWh such as Leeds, Derby, Cardiff, as creating processing nodes  in these cities will dramatically increase the total emissions of the architecture.
Under low traffic rate, 5 fog nodes (Manchester, Glasgow, Newcastle, City Valley, and Preston) which have the carbon intensity of 58 Kg CO 2 e/kWh or less are selected to host applications and serve the users locally instead of moving them to the cloud in order to decrease the network CO 2 emission. Fig. 4 illustrates the optimal applications placement over fog-cloud architecture, considering the emissions values at 10 PM 1 st of January 2020. The results show that 2 cloud locations are selected to host high traffic applications (in Glasgow and Manchester), as they have the lowest emission values (8 and 12 Kg CO 2 e/kWh, respectively). Moreover, 6 cities are selected to serve users locally in the fog nodes, as the emissions in these cities are less than 63 Kg CO 2 e/kWh. For applications with low traffic, 1 cloud and 5 fog locations are selected where the emissions are less than 8 Kg CO 2 e/kWh and less than 58 Kg CO 2 e/kWh, respectively. Similarly, cloud data center serves users from cities with higher emissions (108 Kg CO 2 e/kWh and higher). Fig. 5 shows the optimal applications placement over fogcloud architecture, considering the emission values at 10 AM 1 st of April 2020. The results show that Newcastle is selected to host the sole cloud under both low and high download rates, as it has the lowest emissions (13 Kg CO 2 e/kWh). Moreover, 5 and 4 cities are selected to create fog nodes to  host the applications locally under high and low download rates, respectively. These cities are selected as they have a maximum of 107 Kg CO 2 e/kWh and 73 Kg CO 2 e/kWh, respectively. Fig. 6 illustrates the optimal applications placement over fog-cloud architecture, considering the emissions values at 10 PM 1 st of April 2020. The results show that Glasgow and City Valley are selected to host the clouds under both low and high download rates, as they have emissions as low as 15 Kg CO 2 e/kWh. Five and seven cities host fog nodes, under low and high traffic rates.
Under the low download rate, the emission threshold was 48 Kg CO 2 e/kWh, whereas, under high download traffic, the threshold was 167 Kg CO 2 e/kWh. Fig. 7 shows the optimal applications placement over fog-cloud architecture, considering the emissions values at 10 AM 1 st of July 2020. Similar to the previous scenarios, where a city with the lowest emission value is nominated to host a central cloud, City Valley was selected, as it has only 33 Kg CO 2 e/kWh. Whereas 5 and 6 cities are selected to host fog nodes under low and high download rates, respectively. These cities are selected as they have a maximum of 108 Kg CO 2 e/kWh and 186 Kg CO 2 e/kWh, respectively. Fig. 8 illustrates the optimal applications placement over fog-cloud architecture, considering the emissions values at 10 PM 1 st of July 2020.  Under 5 Mbps download rate, the high traffic justified the creation of fog nodes in all cities. Whereas, under low download traffic, 15 nodes are selected to host fog nodes. These cities have as much as 273 Kg CO 2 e/kWh. Cities with higher emissions are served by a cloud hosted in Newcastle, which has the lowest emissions among all cities (32 Kg CO 2 e/kWh), as the emission cost of transmitting traffic to the cloud in Newcastle through the core network is lower than creating local fog nodes. FIGURE 6. The optimal applications placement of different traffic workload over fog-cloud architecture (0.5Mbps and 5Mbps data rates at 10pm, 1 st of Apr 2020) along with the CO 2 emissions at each city. Fig. 9 shows the optimal applications placement over fog-cloud architecture, considering the emissions values at 10 AM 1 st of October 2020.
Under 5 Mbps download rate, the high traffic justified the creation of fog nodes in all cities. Whereas, under low download traffic, 18 nodes are selected to host fog nodes. Cities with 472 Kg CO 2 e/kWh are served by a cloud hosted in Glasgow, which has the lowest emissions among all cities (87 Kg CO 2 e/kWh), as the emission cost of transmitting traffic to the cloud in Glasgow through the core network is lower than creating local fog nodes. Fig. 10 illustrates the optimal applications placement over fog-cloud architecture, considering the emissions values at 10 PM 1 st of October 2020. Under both download rates (high and low), all applications are offloaded to local fog nodes. Minimum emissions among all nodes, which was observed in Newcastle (with 51 Kg CO 2 e/kWh), did not justify the creation of a cloud data center.

B. CO2 EMISSIONS TRADE-OFF BETWEEN CLOUD/FOG NETWORKS
We can observe from the results stated above (Section V-A) that a trade-off exists between the network emission saved by shortening network journey through replicating applications (VMs) into fog nodes and the extra emission consumed by these replicas (creating more fog nodes). Thus, the creation of an application replica may result in CO 2 emission savings if the former emission exceeds the latter.
For example, in Fig 3, fewer copies of applications with low data rates are replicated over fog nodes. This is due to the CO 2 emissions of creating more fog nodes are higher than the CO 2 emissions of using the network through moving the applications into the cloud. On the other hand, applications with high data rates demand offloads more replicas into the fog nodes to minimize the usage of the WAN network (shortening network journey by using MAN network) and therefore minimize the overall CO 2 emissions of the architecture.

D. A VARIETY OF DIFFERENT DATA RATES ALONG WITH THEIR CO 2 EMISSIONS
The previous results (Section V-A) considered two different data rates; 0.5 Mbps (low) and 5 Mbps (high), which represent the data rate of light web browsing applications (e.g., emails, Google Docs) and Standard-Definition (SD) video streaming applications, respectively. However, there is a variety of different applications that require different data rates, which will be beneficial to broaden the scope of this work. For example, IoT applications only require few kilobytes to update the status of sensors. Furthermore, High-Definition (HD) video streaming applications or online games require higher data rates than light/simple applications.
In this subsection, we take into account these two different types of applications with consideration of two different data rates: 50 Kbps (very low) for IoT applications and 10 Mbps (very high) for HD video streaming applications and online games. Fig. 12 shows the optimal applications placement over fogcloud architecture at 10 AM, 1 st of January 2020, considering these new data rates of 50 Kbps (very low) and 10 Mbps (very high). Generally, when we compare the applications placement shown in Fig. 12 to the similar scenario presented in Fig. 3. We observe that a higher number of application replicas under very high data rate traffic (9 replicas under high data rate demand (Fig. 3) and 11 replicas under very high data rate demand (Fig. 12)). Whereas, under very low data rate traffic (in Fig. 12), we did not observe any application replica in any fog node, as the 50 Kbps (very low) data rate did not justify the creation of any replica (fog node) compared to 5 replicas, under 0.5 Mbps (low) data rate, as shown in Fig. 3. Fig. 13 shows the total carbon emissions of the proposed fog-cloud architecture versus the central cloud and the distributed cloud, while using different data rates than those presented in the previous section (Section V-C). It shows that the proposed architecture can reduce (saving) 98% of CO 2 emissions compared to the central cloud, and 82% compared to the distributed cloud.

VI. THE OPTIMAL APPLICATIONS PLACEMENT OVER FOG-CLOUD ARCHITECTURE -HEURISTIC
The problem of finding the optimum applications placement over fog-cloud architecture that aims to minimize carbon emissions, is a non-deterministic polynomial (NP)-hard problem. For instance, if a is the number of applications and c is the number of locations in fog-cloud architecture, then we will have c x=1 c! (x−x)! possible locations that can be examined to find the optimal placement that results in minimum CO 2 emissions.
Thus, using MILP to solve large problems is not practical. Therefore, we use a heuristic to provide a real-time VOLUME 9, 2021 implementation. Also, the heuristic validates the results provided by the MILP model. Thus, we developed a heuristic algorithm, referred to as a heuristic of the green applications placement over the fog-cloud architecture (HOGAP-FC).  As shown in Fig. 14, the HOGAP-FC heuristic involves two phases of operation: 1) Offline phase: Each node in the network is assigned a weight based on their CO 2 emissions. Thus, we constructed a sorted list of cities from lowest to highest CO 2 emissions and used this list to make cloud/fog placement decisions. In the absence of our sorted list, c x=1 c! (x−x)! combinations will be examined. 2) Online phase: In this phase, the node list obtained from the offline phase is employed to find the optimal placement of applications over cloud/fog architecture. The list of ordered nodes based on the lowest to the highest emissions is examined each day. For example, if the list is {20, 19, 18, 17, 15, 6, 5, 4, 3, 7, 9, 13, 10, 11, 12, 14, 1, 8, 2}, then to place the application in a single cloud, node 20 is the best choice, as it has the minimum emission. If the algorithm decides to have two application copies, then they will be located at nodes {20, 19}. For each application type (specified by its data rate), the heuristic calculates the total emission TE a associated  with placing each application a ∈ App in each placement, J ⊂ E. The total emission is composed of WAN network emission s∈N NE s,a,J , MAN network emission (MAN a,J ), LAN network emission (LAN a,J ), and the cloud/fog emission (CFE s,a,J ), at each placement J and application type a. We use multi-hop heuristic developed in [24] to route the core network traffic between nodes s and d, and calculate the WAN power consumption. After that, the total emission of cloud/fog architecture (TE) is calculated and the associated placement (J ) is selected to host the applications.
The HOGAP-FC algorithm (Algorithm 1) demonstrates the flowchart process of the heuristic (presented in Fig. 14) to find the optimal placement of applications over cloud/fog architecture based on the lowest to the highest CO 2 emissions of each day.

VII. CONCLUSION AND FUTURE WORKS
In this paper, we presented a multi-level approach to achieve a green fog-cloud architecture. We developed a mixed-integer linear programming (MILP) model to optimize applications placement and minimize CO 2 emissions over cloud and fog data centers. The results showed that the optimal locations to host applications are highly reliant on carbon intensity and traffic demands. The results also showed there is a trade-off between CO 2 emission reduced by shortening network journey (minimizing the usage of core network), and CO 2 emission increased by hosting more applications into the fog nodes.
Though the proposed solution is based on the idea of reducing CO 2 emissions in fog and cloud environments, it may be suitable and beneficial for other sectors, such as smart city, self-driving cars, smart grid, and industry 4.0 in order to reduce the environmental impacts caused by these industries.
Different metrics have been taken into consideration in this work, including CO 2 emissions and different data rates to optimize the placement of applications in cloud and fog architectures. Using mathematical modeling, the proposed architecture is compared with the central cloud and the distributed cloud architectures. The model results showed that our proposed approach outperformed both approaches and reduced the overall carbon emissions by 91% and 71%, respectively.
Moreover, we developed a heuristic of the green applications placement over the fog-cloud architecture (HOGAP-FC) algorithm, which showed comparable results to the MILP model.
In the future, we intend to extend the proposed fogcloud architecture by considering cooling operations and their expenses, which are not covered in this work to further study and investigate the capability of the proposed approach.