Orchestration of MEC Computation Jobs and Energy Consumption Challenges in 5G and Beyond

Mobile Edge Computing (MEC) technology philosophy inspires the next generation mobile networks to provide cloud computing capabilities in addition to a diverse range of Information Technology (IT) services with ultra-low latency and higher bandwidth at the edge. One of the most common challenges of 5G-MEC is the management and orchestration across all networks and infrastructure resources as well as end-to-end quality of experience. The decentralized architecture of MEC with independent and non-collaborative servers results in the situation of having underutilized servers with wasted energy. Moreover, the consequences of having highly utilized servers with highly consumed energy are not only the incapability to accommodate all the load of the computing jobs and the dramatic increase in the total OPEX cost, but it also creates some environmental problems. Orchestrating servers’ workload and control offloading the computation jobs is one of the technical advantages of MEC since it satisfies the increasing requirements of modern mobile applications while optimizing the energy consumption and cost. In this work, we consider cluster-based energy-aware offloading framework. The proposed work consists of dual-tier domain divided into clusters of Edge Servers $ES_{s} $ . We have presented the results of our simulation as a proof of our concept that the formulated adaptive strategy to minimize the optimization problem calculation per cluster reduces the energy consumption and enhances the quality of experience while achieving the conservation of the related computing and storage resources cost.


I. INTRODUCTION
The Information and Communication Technology (ICT) touches almost everything in the human life today, including education, commerce, agriculture, environment management etc. The highly increased demand on mobility services and smartphones used to access the cloud services and internet from anywhere at anytime securely has encouraged the mobile service providers to invest more and develop new mobile networks business models that translate the future challenges into solutions.
Fifth Generation (5G) mobile communication technology that introduced high data bit rates and ultra-low latency meets the challenging demands of future compute, storage, database, and other selected Information Technology (IT) The associate editor coordinating the review of this manuscript and approving it for publication was Amir Masoud Rahmani. services, while the sheer ubiquity of delay-sensitive and mission critical applications add more challenges into the 5G.
Mobile Edge Computing (MEC) moves the computing resources and storage capacity services from a centralized cloud to the edge of the mobile network in a partial or full process which analyzes and stores the data closer to the user with real-time processing and enhanced quality of latencysensitive applications. MEC allows the users to deploy virtual machines, containers, managed databases and other IT infrastructure at edge locations; it also reduces the extra network hops to the internet and takes full advantage of the bandwidth and latency advancements of 5G to connect to an application from a mobile device.
The energy consumption of MEC edge server is influenced by multiple components, such as Central Processing Unit (CPU), memory, network interface card, etc. The most two parameters that affect the energy consumption are CPU VOLUME 10, 2022 This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/ and network communication. The server performance is usually presented by its CPU utilization. Server CPU consumes energy even when the server is underutilized or in an idle state as CPU consumes high energy to run in a full utilization state and process complex computing work. Servers also consumes high energy to transmit Protocol Data Unites (PDUs) over the network in a high bit rate. The collaboration between the MEC servers is playing a vital role in orchestrating the computational jobs and energy consumption as the MEC architecture of non-collaborative and independent server will result in the situation of having poorly managed environment with underutilized servers with inefficient use of its energy while other servers are overutilized by unpredicting end-user computation.
The distance between the end-user and server influences the quality of service; MEC is shortening the communications distance by bringing the servers into the edge of the mobile network so instead of sending all data to a central cloud for processing, the network edge analyzes, processes and stores the data with minimum latency.
The increasing number of the servers at the edge in order to provide high coverage and to minimize the application's response time to the end-user requests has enormous advantages, however it results in huge energy consumption and cost.
There are multiple factors that affect the tasks offloading process such as wireless radio connection, end-system hardware configuration, end-user's software configuration, and available bandwidth. Therefore, we have formulated an adaptive algorithm to control this sophisticated process and to support a wide range of use scenarios. In addition, we propose cluster-based energy-aware offloading framework to orchestrate servers' workload and control offloading of the computation jobs to build more flexible, dynamic, programmable and centrally controlled telecommunications network. To the best of our knowledge, the proposed work will add more technical advantages into MEC which satisfies the modern mobile applications increasing requirements while optimizing the energy consumption and environmental impacts intelligently.
This paper contributions may be summarized as follows: • We address the system model and introduce a solution of optimization problem in [1] using a proposed Iterative System Algorithm (ISA).
• We propose cluster-based energy-aware offloading framework that fully automates the collaboration between the edge servers and orchestrates the offloading of computation jobs efficiently in order to fully benefit from the total available energy as well as to minimize the cost.
• We compare the optimization problem size as a main factor affects the computational jobs, versus the number of nodes before and after the clustering framework. The remainder of the paper is organized as follows: section II represents the related work, section III discusses the discussed system model, the proposed system description with system model parameters, section IV illustrates the proposed Simulation discussion and results, and finally the paper is concluded in section V.

II. RELATED WORK
The limited and distributed control of computation jobs, that may results in inefficient resource utilization problem, has motivated the work on optimizing the use of the resources and centrally orchestrated the distribution of the computation jobs. Offloading the computation jobs is one of the technical advantages of MEC. It achieves the modern mobile applications requirements and it optimizes the energy consumption.
For energy-aware MEC structures, optimization problems were presented in [1], [2]. In [1] an optimization problem was formulated to tackle predictions of harvested energy and to predict the cost of grid power needed to be purchased. Considering that offload may occur in regard to saving the power grid cost. However, no clear offloading strategy was discussed in addition to the presented work considers the edge servers power level without taking the actual CPU and buffer utilization into consideration. The discussed system model assumed fully-meshed adjacency sessions between nodes and that adds more overhead to the node's CPU and memory (These issues are taken into consideration in our proposed work). [2] was scoped to two novel optimization problems that formulated to investigate reliable and seamless Network Function Virtualization (NFV) service provisioning. The author in [3] investigated the cost-sensitive service caching problem without resource sharing. [4] presented the study of the architectural models and features of industrial edge computing and virtualization. [5] proposed a reinforcement-learning-based method for offloading making decisions. [6] discussed the research achievements from different perspectives combining edge cloud collaboration and edge intelligence with edge computing security. In [7] a hardware prototype was implemented to provide extensive experiments on a deep neural networks (DNNs) to achieve minimum training time. [8] proposed a model for running application monitoring in an Internet of Thing (IoT) environment. [9], [10] introduced power saving problem on IoT devices. [11] proposed a hand-off model between a specific model named (AGCM) to reduce the waste of energy at both cloud and mobile edge.
It is always motivating to develop new networking concepts, optimize and address the increasing needs of the modern intensive latency-sensitive applications running on battery powered mobile terminals [12]- [14]. There are many challenges of 5G management and orchestration across MEC network infrastructure resources as well as end-to-end service configuration [15]- [18]. Other applications have been discussed to be deployed through the migration procedures in 4G-5G MEC based infrastructure as in [19], [20] and [27]. To satisfy these needs, many researchers discussed these challenges from different perspective.
Therefore, we propose the cluster-based energy-aware offloading framework. The contribution is not only limited to discuss the challenges that affected the MEC environment and how to minimize the negative impact, a solution of the optimization problem using ISA presents the superior performance that increased the accessibility, availability and efficiency.

III. SYSTEM MODEL
The system considers a set of Edge Servers ES s that form fully meshed connectivity with each others. ES s are divided into dual functional tiers as shown in fig. 1. Tier-1 ES s are co-located with the base stations (BS s ), processing the computation and storage jobs that are received from local BS s or from the local domain ES s . Tier-2 nodes are not connected to BS s and handling the jobs that are received from other ES s . The energy source of each ES is a combination of harvested energy and the power grid in addition to the batteries connected to each node. The main target in [1] was to minimize the purchased power grid cost and offload the power level to the other nodes. The assumed prediction model lacks full domain visibility and central offloading control. The offloading algorithm considers the ES power level without taking the actual CPU and buffer utilization into consideration.

A. PROPOSED SYSTEM DESCRIPTION
Our proposed work considers cluster-based energy-aware offloading framework. This work consists of dual-tier domain divided into clusters of MEC ES s . Cluster tier-1 ES s are installed at the edge with the base stations (BS s ) close to the users to process the computation and storage jobs received from the attached BS s or from the other ES s in the local domain. Tier-2 is presented in intelligent MEC orchestrators that are fully-aware of the domain members utilization in order to control offloading the local-cluster members' computation and storage jobs in a programmed manner. Tier-2 orchestrators process computation and storage jobs received from any ES in the local domain.
We started our work with validating the prediction of the purchased energy cost and then applying the formulated optimization problem to evaluate the model in [1] and address the challenges. The model in [1] discussed the offloading strategy with respect to the cost of purchased energy to the system. Our proposed work conquered the observed challenges in [1] by dividing the domain of fully-meshed tier-1 ES s and tier-2 orchestrators into clusters. Each cluster consists of group of tier-1 ES s and tier-2 orchestrators as in fig. 2. Tier-1 ES s establish full mesh connectivity with the local cluster tier-2 orchestrators and other ES s . Tier-1 ES s share its utilization information with the local cluster tier-2 orchestrators. Also receive the updates from the orchestrators regarding the other ES s with available resources and the best path toward it. Tier-2 orchestrators establish full connectivity with the peering clusters tier-2 orchestrators in order optimally exchange the whole domain availability and utilization updates.
Controlling and centralizing the information exchange only through the tier-2 orchestrators reduces the ES s CPU, memory and energy overhead by eliminating the control plane information exchange with other ES s . Tier-2 orchestrators use the learned information to build topological database, calculate the best offloading next hop for each ES and distribute that calculation results with local cluster ES s .
To ensure the high availability, another scenario is also proposed with redundant cluster orchestrators that provides tier-2 orchestrators failure protection. In this scenario, local tire-2 orchestrators will represent one active orchestrator at a time, while it maintains active/standby relationship with each others and synchronize their topological database. Tire-1 will exchange the updates with the active orchestrator only while it will exchange heartbeats only with the standby orchestrators of the local cluster as shown in fig. 3.  which is the sum of harvested energy part saved in battery h i b (k), and for the computation jobs h i c (k) as in (1).
Moreover, the purchased energy g i (k) per each node i is also considered the sum of amount for battery charging g i b (k) while g i c (k) is the energy consumption once charged for computation as in (2).
(3) represents the needed energy per each node defined as c i (k) is related to the jobs r ii assigned to the node i while r ij is the offloading job to node j. Also, c i (k) refers to the sum of the energy per node i as in (4).
The contribution coefficients α and β as assumed in [1] where: α > β, and α > 0, β < 0, The remaining purchased energy or from harvested energy generated from the solar cell attached to ES s is stored in the battery. The b i (k) is initialized with zero saved power. At each time slot k, the available battery power as in (5) is considered as the sum of the available power in the battery from the previous time slot (k − 1) after subtracting the drained energy d i (k) for the same time slot k and the superfluous energy from even the harvested or grid.
η and µ represent the battery coefficients, where η is the percentage reminder of the battery usage. and µ represents the percentage reminder of the battery coming from even the harvested energy and/or power grid. These coefficients are tend to unity as assumed in [1]. (6) to (11) are the system boundaries as follows; While the power in the battery b i (k) is limited to the maximum battery capacity B max i as in (7): The power drained from the battery d i (k) considers less than or equal the remaining power from the previous time slot as per (8): (9) represents the boundary of the harvested energy whether used for computation or saved in battery, could not exceed the total harvested energy H max The sum of offloading jobs r ij (k) from node i to node j minus the number of offloaded jobs r ji (k) from node j to node i is limited to the number of local computed jobs r ii (k) to the same node i at each time slot k as per (10) j∈N \i (r ij (k) − r ji (k)) ≤ r ii (k), ∀i, ∀k.
The number of local processed jobs even received on the same node or offloaded from another nodes should be less than or equal to the maximum number of jobs R max ii (k) to be processed per time slot k as in (11) The optimization problem in (12) minimizing the equations from (1) to (11).
where ψ = PTC (Power Total Cost) that formulated the power cost in the following equation:

IV. SIMULATION DISCUSSION AND RESULTS
To build the experimental setup, we have used computerbased MATLAB 2016 simulation on a computer with Intel i7 Quad-core 3.2 GHz CPU and 64 GB RAM to evaluate the proposed model performance. We implemented this environment by considering a group of ES s that use input power as harvested energy, power battery and a power grid. The optimization problem in [1] is formed and tested over 360 hours. In our simulated environment, we have built similar setup over 360 hours however we have achieved nearly predictive energy output in the first 24 hours followed by consistent values. We focused on comparing our proposed model and the discussed model performance within the first 24 hours to evaluate the proposed ISA algorithm with focusing on the outcome orchestrated clusters approach.

A. SIMULATION DISCUSSION
In our work to measure and solve the optimization problem, we initialized the parameters for i * w for the set N of nodes where i represents any node and w is the prediction window size. The considered time horizon is 24 hours. Each node is establishing fully meshed sessions with all other nodes resulting in total number of sessions N * (N −1)/2. The main challenge was how to minimize the purchased energy. The lab configuration of 10 nodes in a fully meshed topology is implemented in 170 configuration row. The extensive number of established sessions made it a sub-optimal solution as it is directly proportional to the number of nodes. Changing the flat topology into clusters with local orchestrators has minimized the optimization problem size as the configuration rows decreased to 22 rows in our lab test. The tier-1 nodes establish sessions with local cluster members only, which makes it more scalable solution. Although considering the high availability and failure protection required adding a secondary orchestrator to each cluster, it did not add high processing overhead. Moreover, increasing the number of nodes does not affect the optimization problem size as in flat topology.

B. SIMULATION RESULTS
Our work begins with validating the discussed work in [1] on different prediction window starting with the first prediction window w at k = 1 hour with increments of two. The discussed work consists of 5 constraints (first 5 equations) and 6 boundaries. For each node i in time slot k, these variables are solved by applying ISA into the optimization problem as mentioned. Table 2 presents the simulation applied parameters value.
In [1] the energy cost should be predicted 1 day ahead to estimate when is the best time to purchase energy. [1] assumed that each node i is able to offload to any of other j nodes in the fully meshed flat topology; the domain size is directly proportional with the optimization problem size. In our work, the orchestrators are fully aware of the whole domain members' energy utilization. Each orchestrator establishes fully meshed control plan sessions with all other orchestrators in the local domain. That allows to exchange the utilization and available energy information and create a full picture of the whole domain resources utilization. The orchestrator will instruct the local cluster member to offload its excess computation jobs to the nearest available domain member. The following Algorithm 1 illustrates the non-linear optimization problem solution for the cluster-based energyaware offloading Framework. In the algorithm, we created an array cycled with two for-loop one started from w = 1 with step 2 till w = 24 and the other loop for moving the window till reach 360 hours. Reference Constraint function 5: Reference min function 6: call F mincon 7: add F val to the sum 8: Reference the battery stored energy 9: end for 10: Save the purchased energy cost 11: for W = 2:2:24 do 12: for J = 1:360 do Solve the optimization problem for the given duration 13: end for 14: Divide this sum by the sum above to get the saving percentage 15: end for 16: Plot the results with respect to W In the following fig. 4, we evaluated the prediction of energy cost in percentage per time and compared the results of [1] with our proposed work results across different w. ISA algorithm is implemented as a solution to the discussed nonlinear optimization problem and the results are nearly the same after applying the ISA over w within the 24 hours.
With the increments of k, moving from time slot to the next time slot, a new prediction calculation should start to solve optimization problem for the new time window parameters. We compares the running simulation with the discussed model within the first 24 hours however the simulation for T sim = 360 hours to discuss the overall system performance. Fig.5 represents the optimization problem size versus the number of nodes. These nodes are distributed in a fully meshed flat design without cluster formation. ISA algorithm is applied to solve the optimization problem. As shown, increasing the number of nodes increases the optimization VOLUME 10, 2022  problem size which consumes more CPU and memory resources.
In our lab simulation, we started with the assumption of 10 nodes in a fully meshed topology over different prediction window w = 1, 12 and 24. we ran the optimization problem on different number of nodes from 10 to 100 nodes. As shown, increasing the number of nodes over different prediction w enlarges the optimization problem size. Based on this observation, we noticed that with the increasing number of fully meshed nodes will also increase the optimization problem size that impacts the computational tasks and the amount of resources consumption.
In order to minimize this impact on the computation resources, we proposed the cluster formation model where each group of nodes is segregated in a cluster to segment the optimization problem into smaller optimization problems. The optimization problems will be addressed within any formed cluster over different w. To evaluate the cluster formation, we measured the problem size when the number of nodes is 16 nodes (where we assumed that the cluster could formed from the relation [(cluster formed from √ N ) where the minimum number of nodes to form a cluster is two nodes.] and measured the optimization problem size with the increase of N up to 100 nodes. Fig.6 shows the impact of cluster formation when tested with different number of nodes. However we started with 10 nodes (we set the node 16 is our reference just to facilitate the comparison with and without forming the cluster).    Fig.7 illustrates a comparison between the optimization problem size with respect to the number of nodes at different w. We evaluated the optimization problem size versus the number of nodes with and without forming the clustering. As seen from the comparison, the processing done on solving the optimization problem is positively affected with cluster formation which in turn enhance the overall processing utilization.
To discuss the impact of forming the clusters, a cross section taken at estimated number of nodes = 64 nodes with respect to different w as shown in Fig.8. We observed dividing the optimization problem size into smaller problems to be solved in each cluster which minimize the computations and processing used to solve the main optimization problem and that enhancing the overall system performance.
The following table 3 represents the estimated saving percentage of processing the discussed optimization problem. As per the results in table 3, the optimization problem size dramatically decreased at different w. As shown the saving percentage stating from around 12 % (at the randomly selected N = 64) to reach around 70 % saving in the processing. Forming the clusters minimizes the optimization problem size which in turn minimizing the overall processing computation and the overall energy consumed.

V. CONCLUSION
The proposed MEC cluster-based energy-aware offloading framework that consists of dual tier domain divided into clusters of ES s and central orchestrators provides more efficient 5G-MEC workload orchestration, energy consumption and cost management. It is designed to operate efficiently in a highly scalable domain of computation resources as well as large workloads. We address the system model and introduce a solution for the discussed non-linear optimization problem using the proposed Iterative System Algorithm (ISA). Our solution considers sourcing the power from energy harvesting system, energy batteries in addition to the power grid. The simulation outcomes illustrate the positive impact of cluster formation and intelligent orchestration layer on minimizing the optimization problem size while satisfying the environmental concerns. The proposed model demonstrates a considerable enhancements that lead to save the overall computations and processing with percent more than 50 % by orchestrating the computation jobs within the 5G(and beyond)-MEC architecture in cluster-based model.