Hybrid Workflow Scheduling on Edge Cloud Computing Systems

Internet of Things applications can be represented as workflows in which stream and batch processing are combined to accomplish data analytics objectives in many application domains such as smart home, health care, bioinformatics, astronomy, and education. The main challenge of this combination is the differentiation of service quality constraints between batch and stream computations. Stream processing is highly latency-sensitive while batch processing is more likely resource-intensive. In this work, we propose an end-to-end hybrid workflow scheduling on an edge cloud system as a two-stage framework. In the first stage, we propose a resource estimation algorithm based on a linear optimization approach, gradient descent search (GDS), and in the second stage, we propose a cluster-based provisioning and scheduling technique for hybrid workflows on heterogeneous edge cloud resources. We provide a multi-objective optimization model for execution time and monetary cost under constraints of deadline and throughput. Results demonstrate the framework performance in controlling the execution of hybrid workflows by efficiently tuning several parameters including stream arrival rate, processing throughput, and workflow complexity. In comparison to a meta-heuristics technique using Particle Swarm Optimization (PSO), the proposed scheduler provides significant improvement for large-scale hybrid workflows in terms of execution time and cost with an average of 8% and 35%, respectively.


I. INTRODUCTION
The Internet of Things (IoT) is a technology that refers to a large set of objects (machines, devices, etc.) that can connect and share data without requiring human intervention. The adoption of this technology steers innovation in the form of smart and intelligent applications to simplify human life and enables new business-oriented, user-specific, and humancentric applications. The next generation of IoT platforms will be applied in domains like 1) industrial for business continuity management and collaborative supply chain management [1], 2) public services in smart cities and social sensing [2] and 3) customer services for autonomous cars and smart homes [3]. In IoT, the large network of connected physical devices of sensors and actuators can produce a massive amount of data that need to be handled in real-time or near The associate editor coordinating the review of this manuscript and approving it for publication was Chih-Min Yu . real-time. Moreover, IoT devices generate data with different frequencies and different timeliness requirements for data delivery. Thus, the IoT-generated data possesses properties that conform to the big data paradigm [4]. However, IoT big data is difficult to process in traditional computing systems. Data stream is not available at once, the stream rate is often of high speed and may vary with time, and timely analysis of the data stream is critical [5].
Data analytic applications are commonly managed and executed with workflow technology. Generally, a workflow is a systematic representation of a process as a set of dependent tasks accordingly to a set of rules [6]. The workflow model aims to automate and minimize the complexity of managing various types of processes, such as human activities, business processes, and scientific experiments [7]. In data analytic applications, it is common to integrate batch and stream processing models [8]. Batch processing involves storing the upcoming data before processing while streaming VOLUME 9, 2021 This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/ processing is related to performing operations on data streams in a real-time or near-time manner. We refer to this integration as a hybrid workflow [9]. Hybrid workflows can be applied in many application domains like traffic monitoring [10], social sensing [11], and business analytic [12]. Stream and batch processing have different Quality of Service (QoS) requirements. Stream processing is latency-sensitive and subjects to constraints like stream rate and throughput [13]. Data stream arrival rate may change over time making it hard to estimate data stream collection and processing intervals. Thus, it's not trivial to determine the size of these intervals in advance. If the interval is too large, the accuracy of prediction can deteriorate when the nature of the data changes, and if the interval is too small, accuracy can deteriorate when that is rather stationary. On the other hand, batch processing has a less time sensitivity processing and more computation-intensive to conduct large data computation for data aggregation and predictive modelling functions. Overall, both processing models aim for maximizing throughput, which determines the efficiency of processing computation under certain user and application QoS requirements.
Cloud computing offers a robust and scalable computation model. However, the bottleneck of pushing continuous data stream from a large number of IoT devices cannot be neglected for latency and cost constraints [14]. Transmitting large chunks of data requires a high network bandwidth to reduce data migration latency. Moreover, much of the raw data (from IoT devices) is unnecessary or unusable and cannot be directly injected into computation cycles. As a result, reducing transmitted data to the cloud and performing more analytic closer to IoT devices is convenient. This refers to edge computing which is a general term of enabling technologies allowing computation to be performed at the edge of the network [15]. However, this may increase the complexity of managing and monitoring the execution of IoT-based workloads (such as hybrid workflows) on such a heterogeneous computing system, i.e., edge cloud. Hybrid workflow scheduling implies an understanding of the differences in processing behaviour of stream and batch applications to propose workflow schedulers that support maintainable integration between these applications with consideration of user QoS constraints.
Few research works have studied the hybrid stream-batch workflows and proposed general-purpose execution models [16]- [19] to maintain QoS constraints like latency, throughput, fault-tolerant, etc. However, the hybrid modelling is not well-formulated to reflect the data and computation dependency between stream and batch tasks. The motivation of hybrid model is to provide an efficient management and control over applications which are complex in scale and internal dependencies, involve short-term stream intervals and online batch feeding, and flexible to their parameters tuning. In this paper, we are extending our previous work [9] to propose a hybrid workflow scheduling framework on edge cloud computing that considers the integration requirements for hybrid workflows while optimizing the workflow execution time and monetary cost. The framework involves algorithms for resource estimation, provisioning, and task scheduling. This paper has the following contributions: • Hybrid workflow resource estimation algorithm based on the gradient descent search (GDS) technique.
• Cluster-based resource provisioning and denting adjustment scheduling algorithm for hybrid workflow on edge cloud computing environment.
• Comprehensive performance analysis of estimation and scheduling algorithms, including scheduling adaptability, edge capability, and optimization time Next, we discuss some of the state of the art algorithms and techniques for resource estimation, provisioning, and task scheduling in the cloud and edge cloud systems are presented in Section II, and then the problem formulation is described in Section III, including the edge cloud system model and a detailed description of the application model. The hybrid workflow scheduling in the edge cloud system is described in Section IV and the system performance is evaluated based on experimental findings from various perspectives in Section V. Finally, conclusions are provided in Section VI.

II. RELATED WORK
The main objective of workflow scheduling is to generate scheduling plan(s) that maximize efficiency under certain optimization objective(s) such as makespan, execution time, monetary cost, reliability and resource utilization. Directed acyclic graph (DAG) is commonly used to represent a scheduling plan; vertices denote workflow tasks, and the edges show the dependencies among them [20]. Workflow and DAG are often used interchangeably in the literature. Generally, a workflow is a systematic representation of a process as a set of dependent tasks accordingly to a set of rules [6]. The workflow model aims to automate and minimize the complexity of managing various types of processes, such as human activities, business processes and scientific experiments [7].
However, few researchers have proposed models which are efficient and relevant to hybrid workflows. This section provides a broad analysis of the research body on workflow scheduling, including resource provisioning and task allocation in different computing systems.

A. RESOURCE PROVISIONING AND TASK SCHEDULING IN CLOUD COMPUTING
In cloud computing, resource provisioning is an adaptive process of provisioning and deprovisioning resources according to workload changes and service demand at a certain point in time [21]. Resource provisioning is critical to control resource utilization and cost by avoiding resource over-provisioning. Provisioning is estimated by reactive or proactive mechanisms. Reactive mechanisms continuously track service workloads and fire scaling triggers in response to resource demands. However, the time needed to update and apply a new provisioning plan can be detrimental, particularly for latency-sensitive applications that involve real-time or near real-time processing. Proactive mechanisms are efficient in incorporating timely-constrained provisioning by observing workloads and predicting resource demands by applying statistical or mathematical models, such as queuing theory, reinforcement learning and control theory [22].
Mao and Humphrey [23] proposed two algorithms to resolve the auto-scaling issue: scheduling-first, and scalingfirst. The first algorithm applies total budget distribution to each workflow task, generates the fastest execution plan, and finally acquires cloud resources. The second algorithm estimates the required resources concerning data size and resource capabilities, and finally schedules the workflow tasks. Nikravesh et al. [24] proposed a predictive auto-scaling system to scale cloud resources automatically for different types of workloads and fixed observation window size. The work does not illustrate how workload features are injected into the prediction model. Wagner and Michalewicz [25] worked on time series data prediction based on an adaptive sliding window. The technique is not suitable for time-sensitive applications due to the complexity of real-time windows size identification. Deypir et al. [26] used an adjustment technique for stream arrival rate with variable window size. Experimental evaluations showed the proposed technique efficiency in adapting window size to improve system performance. However, the technique involves user intervention to set the window size reduction threshold. Warneke and Kao [27] considered the dependencies between application tasks, and determined the specifications of cloud resources needed to provide efficient resource allocation and scheduling framework. However, application modelling does not study how changes in workflow structure or input data are reflected in generated schedules.
For task scheduling in cloud systems, the literature has demonstrated an extensive research body which covers a wide range of task scheduling aspects for optimizing objectives like cost, execution time, energy, reliability, security, energy, etc. [28]- [30]. Topcuoglu et al. [31] proposed one of the best heuristics scheduling algorithms, the heterogeneous earliest finish-time (HEFT) algorithm to minimizes the overall workflow makespan through minimizing the earliest finish time for critical tasks. HEFT performs task allocation in two steps: task prioritizing and instance selection. Task prioritizing ranks tasks in a list according to the cumulative execution time on each VM instance and average communication time between VMs of dependent tasks. The unallocated task with the highest rank is selected and mapped to its best instance, which guarantees the lowest finish time. Durillo and Prodan extended the HEFT algorithm [31] by proposing the Multiobjective-HEFT (MOHEFT) algorithm [32]. MOHEFT is a generic multi-objective scheduling algorithm, which works by generating several scheduling solutions. The quality of a solution is assessed using the metric of crowding distance. However, complete coverage traversing is adopted in MOHEFT to generate new solutions for assigning tasks to instances, which consumes a large amount of time. Thus, the technique is not efficient for large-scale workflows [33].
Abrishami et al. [34] adapted the partial critical path (PCP) algorithm for deadline-constrain. The algorithm works by generating PCPs, and for each PCP, included tasks are allocated to the same VM instance based on heuristics. Grouping critical tasks can enhance resource utilization and reduce the total execution time while meeting the deadline. The technique can reduce computation time and cost, but the communication time and cost are not considered.

B. THE CONTRIBUTION OF EDGE COMPUTING
In cloud computing, data transfer times can maximize the overall workflows execution time and cost. IC-PCP overcomes the issue by grouping and scheduling dependent tasks on the same VM to reduce the amount of transferred data between VMs [34]. However, IC-PCP does not provide an accurate estimate of execution and transmission time. Sahni and Vidyarthi [35] applied the grouping strategy by constructing pipelines of interdependent tasks aiming to reduce the workflow execution cost while meeting the deadline. However, no priority policy is applied to select the next allocated pipeline [44]. Lin and Wu [45] proposed a technique to minimize the overall workflow execution delay considering the budget constraint. The scheduling problem assumes a one-to-one VM allocation to each workflow task. However, latency is a bottleneck for running stream applications on cloud systems. Issues like streaming from devices over long distances with cloud resources and the unpredictable quality of transmission networks need to be considered in stream workflow scheduling. However, in reality, migrating streams from the smart objects and transferring the data to the centralized cloud environment is considerably not efficient due to the increase in the latency time and response time of the real-time data items.
Computing models like edge computing and edge cloud computing are proposed to resolve such issues [46]- [50]. Edge computing is intended to overcome the challenges (limited bandwidth and network latency) of migrating large amounts of data for real-time data processing. Recently, many models have been proposed to conquer the challenges of stream workflows by adopting resource provisioning and task scheduling techniques for use in an edge cloud computing environment, to improve edge resource utilization, reduce latency by processing data-intensive tasks in nearby edges, maximize communication stability for high-performance stream processing, and provide an efficient task placement strategy to achieve reliable resource provisioning.
Madej et al. [37] proposed a fairness-based scheduler for edge cloud computing. The paper compares four scheduling techniques, namely, (First-in First-out) FIFO, a client fair, priority fair, and a hybrid that accounts for the fairness of both clients and job priorities. The experimental results demonstrate that the hybrid technique is the best and that the fair scheduler is feasible to implement in edge cloud systems. Naha et al. [38] proposed deadline-constrained resource allocation and provisioning in the fog cloud environment. The algorithm addresses the issue of user QoS variation and resource limitations on fog computing. Resources are allocated based on a scoring system and considering various parameters. Zhou et al. [39] proposed an online gradient descent technique to estimate the rate of data streams to reduce the cost of cross-edge IoT data streaming processing. The work does not consider tuning other streaming processing parameters like aggregation processing interval and throughput. Farhadi et al. [41] proposed a data-intensive application scheduler using a mixed-integer linear program (MILP) on an edge cloud system. The technique aims to maximize the system utilization in terms of the number of placed scheduled services. Zhang et al. [42] adopted a deep reinforcement learning (DRL) framework for joint workflow scheduling optimization on a hybrid edge cloud system. Wu et al. [43] also applied a DRL framework to schedule IoT stream applications on an edge system while reducing the execution latency.

C. HYBRID WORKFLOW SCHEDULING
Hybrid workflow, or batch stream combination, has been sufficiently studied and many research frameworks have been proposed to address some of the requirements of such an integration. Kailasam et al. [16] proposed BStream which combines batch and stream processing at different cloud layers and uses a predictive model to estimate resource allocation and task distribution to meet task deadlines. Carbone et al. [17] developed Flink, so-called Apache Flink, a unified stream and batch computing framework which expresses various classes of data processing applications as fault-tolerant dataflows in a low-latency and high throughput. Apache Flink is designed for ''stateful streaming processing'' which handles batch tasks as a bounded stream resolving the high latency of micro-batch architecture in some streaming processing frameworks like Spark [18].
Zhang et al. [36] proposed an online heuristic algorithm called DyBBS for adaptive batch size and execution parallelism to reduce end-to-end processing latency and reflects the workload changes. Pishgoo et al. [19] proposed batch-stream processing architecture for distributed modeling of streaming events to detect anomalies in such events. They argue that there is a limitation on considering the hybrid architecture regarding how different processing units should interact with each other specifically at the algorithm level. Huang et al. [40] proposed a linear optimization model to schedule IoT batch tasks in a mobile edge computing (MEC) system. Meanwhile many industrial frameworks demonstrate adequate support for hybrid workflows, but the influence of workflow structure, i.e., the dependency between stream and batch tasks (in terms of data and computation), is not well formulated to reflect workflow execution optimization for cost, throughput, deadline, etc. Thus, the need arises for developing schedulers that can adopt the hybrid workflow execution behaviour. Table 1 summarizes the most related works in terms of application type, optimization technique, objectives and computing system. The current literature does not effectively illustrate the opportunities of edge cloud systems in resolving scheduling issues of hybrid workflows like load balancing, edge capacity limitations and data communication between edge and cloud resources. In this work, we provide an end-to-end hybrid workflow scheduling framework in an edge cloud computing environment. The framework handles the requirements of integrating stream and batch processing in IoT applications. The framework is relied on proposed algorithms and techniques for resource estimation using gradient descent search (GDS) technique and resource provisioning and task scheduling using a cluster-based approach.

III. THE SYSTEM MODEL
Cloud computing is not the ideal computing system for latency-sensitive applications, such as real-time gaming, augmented reality and real-time streaming [51]. Because cloud resources are located closer to the core network, the round-trip latency of these applications will be high due to passing data through multiple gateways. Edge computing refers to technologies that permit the computation at the network edges, and act as an intermediate layer to transmit streams to cloud services on behalf of IoT devices [15]. However, a high percentage of this data is temporary and only a small amount might contain meaningful information. Edge computing has a significant role in processing massive data and enables better filtering of what data is uploaded to the cloud. For instance, an autonomous vehicle requires a huge amount of data, in the forms of images and video, to be processed in real-time to make safe driving decisions. However, the availability and low efficiency of edge servers are decisive factors in quality achievement [39]. Hybrid workflow specifications require the development of efficient resource provisioning and scheduling techniques that coordinate the execution of hybrid workflows on edge cloud systems.

A. EDGE CLOUD COMPUTING SYSTEM
The edge cloud system is convenient for hybrid workflows because the stream tasks with latency sensitivity can benefit from the availability of edge resources, whereas batch tasks with heavy workloads can be processed at powerful computation nodes in the public cloud. Figure 1 shows the adopted edge cloud system model which is composed of IoT, edge and multi-cloud resource layers. The three layers are explained as follow: • Layer 1 (IoT). The layer represents the user interaction layer, in which IoT devices (sensors, smartphones, relays, etc.) generate data which can be on various formats and bounded to a particular point of time. IoT devices can send workloads to the nearest edge nodes or cloud resources.
• Layer 2 (edge layer): This layer represents all devices on the path between the IoT layer and cloud layer. An edge node can be a non-stationary computation device, such as mobile devices or a Raspberry Pi, or a stationary device, such as a personal desktop, company server or a cloudlet. We assumed that edge devices are limited in computation and storage capabilities, but able to handle some pre-processing tasks on the stream processing pipeline. Overall, the edge layer offers computation close to data sources (IoT devices and sensors) to reduce data transfer time, and performs pre-processing in a timely manner constraint [15].
• Layer 3 (multi-cloud services): The third layer includes cloud resources that have high computational and storage capabilities, and can handle heavyweight computations such as predictive analysis, machine learning, business intelligence, big data analytics, and complex visualization. Cloud providers offer a range of computing types including Virtual Machines (VMs), Containers and Serverless computing.
. . , t n } with direct edges E and no cycles or conditional dependencies. A task t i represents a stream (t i S ) or batch (t i B ) task application. The two types of application are fundamentally different in their modelling features. Table 2 presents mathematical notations used in the hybrid workflow modeling.
A stream task t i S represents the execution of a stream processing pipeline and it is modeled as an M /G/c queuing system [52]. The queuing modeling provides an estimation of the amount of resources to run a stream processing pipeline under constraints like system utilization, waiting time, and system throughput. Stream processing system must consider these constraints with the variation in workload to achieve a highly optimized, minimal overhead execution engine to deliver real-time response for high-volume applications [53]. The number of servers c in a queuing system refers to the level of parallelism where processing cycles are replicated in case of a high stream arrival rate. A stream task t i S is modeled as: Based on the approximation by Kleinrock [52], the number of queuing system servers c i is estimated as follow: To reduce the complexity of M /G/c queues, we assume that at a given moment, all servers c will be busy. Hokstad [54] showed that we an M /G/c queue which has an identical service time (S = 1/µ) can be represented as a M /G/1 queue with service time (S = 1/µc). Based on this assumption, the approximation provided by Kleinrock [52] is adopted to estimate the waiting time on queue W q : A batch task t i B has the following features: A batch task t i B receives data collectively from stream and batch tasks. The total amount of data L i received by a batch task is computed based on the following formula: where d j is the amount of data generated from task t j , which can be a stream or a batch task. The number of servers c i (parallelization factor) to run a single iteration of a batch application is estimated as follow:

IV. HYBRID WORKFLOW SCHEDULING IN EDGE CLOUD RESOURCES
Workflow scheduling is utilized for mapping tasks to computation resources and planning their execution while fulfilling dependency between tasks and achieving certain QoS parameters and resource preservation goals. Without loss of generality, Figure 2 shows hybrid workflow scheduling framework which consists of three steps, including resource estimation, resource provisioning, and workflow scheduling.
In the resource estimation stage, the amount of resources and time to run the workflow are estimated under task execution constraints, including deadline, throughput, and waiting time. Accordingly, workflow tasks are formulated in execution groups. Next, the required computation nodes at the edge cloud system are provisioned to host the execution of these groups. Two type of resources are involved, namely, public clouds (D 1 and D 2 ) and edge nodes (D 3 ). Lastly, the workflow tasks are mapped to provisioned resources considering a multi-objective scheduling optimization of execution time and monetary cost.

A. RESOURCE ESTIMATION WITH GRADIENT DESCENT SEARCH APPROACH
The optimization objective of our framework aims to minimize the total execution time E and the number of computation units (cores) R while meeting the execution constraints of stream and batch tasks. The estimation process works by grouping workflow tasks to construct an execution plan considering their execution dependencies and maximizing resource utilization at the group level. The group-based technique can simplify resource provisioning and task scheduling in an edge cloud environment, particularly for complex hybrid workflows. The optimization function for resource estimation is formulated as:

1) RESOURCE ESTIMATION PROBLEM FORMULATION
Gradient descent (GD) is a first-order iterative optimization technique of finding a local minimum for linear and non-linear functions [55]. Gradient descent comes in three variants: batch or vanilla gradient descent, stochastic gradient descent (SGD) and mini-batch gradient descent. Based on the minimum of data required to perform the estimation process and the low complexity of the estimation model with pre-defined workflow execution functions, the batch gradient descent variant is used [56]. The gradient descent search (GDS) process starts at some arbitrary point x (0) and then iteratively moving at direction x (k) after every step k ≥ 0 by step size γ k to the next point x (k+1) as: Since the objective is minimizing the function, the step size γ i value should be selected in a way that will minimize the function value of the new point such that the step size γ * k at step k satisfies: The application of the gradient descent search (GDS) on the resource estimation problem is described as follow: 3 refers to the number of problem parameters (λ i , ω i , τ i ) of a stream task t i S and d refers to the number of stream tasks.
• The workflow execution function f at point (workflow setup) x (k) is written as: where E(x (k) ) and R(x (k) ) are the workflow execution time and amount of resources, respectively. This reflects the resource estimation optimization objective provided in Equation 7.
• The estimation process is a constrained GDS problem as it relies on constraints such as minimum throughput τ i and maximum deadline ϑ i .
• The optimization problem objective is to find the matrix x ( * ) in which the function has a local minimum and mostly a global minimum (as dealing with convex workflow execution functions).

2) RESOURCE ESTIMATION ALGORITHM
The estimation algorithm includes a set of steps to find the optimized estimation cost (number of cores R) with minimum execution time E. Next, The Constrained-based Gradient Descent Hybrid Workflow (C-GDHW) resource estimation algorithm is described in algorithm 1. The algorithm receives three inputs including 1) workflow structure w, 2) an initial solution x (0) which represents a 3×d stream task configuration matrix. As shown in Equation 10, the configuration matrix presents the execution setup for all stream tasks including the arrival rate λ, the window size ω of collecting and processing incoming streams, and the percentage of stream of data to be processed or throughput τ . The window size, so-called batch sizing in Spark Streaming [18], is the time duration to create a batch input and send it to a batch queue. Zhang et al. [36] experimented stream parameters tuning and their results illustrated the significance of an adaptive batch streaming processing control algorithm for reducing the end-to-end application processing latency.
At each iteration k, the algorithm performs the following steps: 1) Based on the current solution x (k) , the workflow execution cost f (x (k) ) is estimated according to Equation 11. The cost estimation flow is described in lines 4-18. At line 4, the callculateTaskTimes is called to calculate the execution times for all workflow tasks including the EST , which is the earliest task starting time and the LFT , which is the maximum execution finish time. 2) Based on task execution times, at line 5, the given solution is validated to satisfy the following constraints for a stream task t i S (Equations 12 and 13) and for a batch task t i B (Equation 14): where ω i and λ i are minimum values of window size and arrival time, respectively.
where ET i and E i are processing time, and task execution time, respectively. A batch task processing time ET i (as provided in Equation 15) is the total time required to process the total amount of received data L i based on task service rate µ i . Accordingly, the task execution time E i (as provided in Equation 14) is computed, such that, it does not exceed the task deadline ϑ i and maintain the least amount of resources c i .
3) If the solution is valid, the grouping process for workflow task V is started. The algorithm tries to fit the task t to a group in the group list G in respect to a certain level of deadline relaxation, otherwise, a new group is created. 4) Once all tasks are assigned to execution groups, the cumulative execution time E and the total number of cores R are calculated in lines 15-19. Accordingly, the cost function value f (x (k) ) at the given solution is computed based on Equation 11. 5) At line 21, the algorithm convergence is checked against the threshold ∂. If not converged yet, the gradient value g (k) is computed and the next step (solution) x (k+1) is generated as presented in lines 24 and 25, respectively. Lastly, the best workflow configuration x (best) is returned. 6) The time complexity of the C-GDHW algorithm is O(kn 2 ) where k is the cost of computing the function derivative and n is the number of iterations for the function to converge [57]. 7) The space complexity of the C-GDHW algorithm is O(nd + d) where d is the number of tasks and also can be the number of groups in the worst case.

B. HYBRID WORKFLOW PROVISIONING AND SCHEDULING ON EDGE CLOUD COMPUTING
The next step after estimating the required amount of resources is to provision them on the edge cloud environment and then map the allocated resources to workflow tasks. In this work, we assume prior knowledge about the incoming workload, i.e., the amount of data to be processed and transferred during the execution cycles. Thus, offline scheduling is a preferred option [58]. The workflow scheduling is formulated as a multi-objective optimization problem to jointly reduce the end-to-end workflow execution time T and monetary cost C. Both the workload computation and data transfer costs are included in the optimization process, which approaches to achieve the following objective function: A Cluster-based Hybrid Workflow Provisioning and Scheduling (C-HWPS) technique is adopted to meet the requirements of hybrid workflow scheduling to provision demanded resources and schedule tasks in an edge cloud computing system. The cluster-based technique works by constructing sequential execution paths where each path represents the execution pathway of dependent tasks which should lead to the last workflow task. As a result, one task can be founded on different execution paths. Figure 2 Algorithm 2 Cluster-Based Hybrid Workflow Scheduling procedure C-HWPS(G, D) provides an example of the execution of three paths based on the input workflow. Three execution paths are constructed {P 1 , P 2 , P 3 }.
The C-HWPS technique is an iterative process of provisioning and scheduling tasks on edge nodes D 1 and two public clouds D 2 and D 3 based on the selection of an execution path at each iteration, which has the minimum of the objective function (T * C). The execution time T is the sum of computation time (for unscheduled tasks) and the data transfer time (between scheduled and unscheduled tasks). Similarly, execution cost C (processing and data transfer) on edge cloud resources is computed. The scheduling process will continue by adding paths and optimizing time T and cost C until allocating all tasks.
Algorithm 2 presents the cluster-based hybrid workflow resource provisioning and scheduling which aims to minimize the objective function T * C. The algorithm takes the inputs of task groups G as the outcome of the resource estimation phase and the current status of the edge cloud system resources D. The status provides details about resources capacity, availability, connectivity and cost. At line 4, the algorithm calls the findExecutionPaths function to construct workflow execution paths P based on tasks dependencies structure.
Next, the algorithm iterates over execution paths P to find the one which has the minimum optimization value of the current iteration T iter * C iter where execution time T and cost C on edge cloud system D. The algorithm starts with the first unallocated path u which has at least one unscheduled task, in other words, return T , C, S scheduled tasks. Unscheduled task set [u − A] computation is evaluated in the edge cloud system D using the function computePathInEdgeCloud, line 11, which calls the routine provided in Algorithm 3. In addition to the optimization values, the function returns the associated resource provisioning and scheduling plan S on the edge cloud system D. The path with minimum time and cost p target will be selected and scheduled based on the resource provisioning and scheduling plan S target . Finally, the algorithm returns scheduled tasks A, and optimization object variables T and C. The time and space complexity of the C-HWPS algorithm is O(n) where n is the number of execution paths and also can be the number of tasks groups in the worst case. Algorithm 3 describes the resource provisioning and task scheduling steps for executing a path p in an edge cloud system. This work does not assume collaboration between edge and cloud resources. Thus, the scheduler will effectively try to keep the execution path unity in one computing system and reduces the amount of data exchange at different computing layers. The algorithm provides two main functions. The first is to schedule tasks in either edge or cloud, lines 4 and 7. Based on the available resource configuration, execution time and cost are calculated. Stream tasks have the priority for scheduling at the edge layer. The second function computeTransferTimeAndCost, lines 5 and 8, computes the data migration time and cost computing nodes based on the data transmission (bandwidth) quality and cost. Finally, the algorithm returns the resource provisioning and scheduling plan S with the expected execution time T and cost C for the given execution path p. The time and space complexity of the Algorithm 3 is O(1) as execution time and cost calculations are performed at constant time.

V. PERFORMANCE EVALUATION
In this section, we evaluate the performance of the proposed resource estimation technique, C-GDHW, against two nominated and comparable techniques. The first technique is a meta-heuristics technique using Particle Swarm Optimization (PSO) proposed in our previous work [59] and the second one is the full-mode technique which conducts a semi-optimization process at queuing system utilization level with no reduction mechanism to be applied for stream arrival rate and window size. This technique works as a baseline to assess the performance parameters' tuning of the other two techniques. In addition, this section evaluates the contribution of edge capability in optimizing the execution of hybrid workflows. Finally, the optimization time of the proposed algorithm is investigated in comparison to the PSO-based algorithm.
Next, we provide the experimental setup which includes building various hybrid workflow models from an existing IoT application benchmarking and the resource configuration of the edge cloud system.

A. EXPERIMENTAL SETUP
For the purpose of experimenting with hybrid workflow scheduling, we used IoT dataflow benchmarking results provided by Shukla et al. [60] to build three types of workload models that satisfy the design concepts of hybrid workflow with a reasonable level of structural complexity and data dependency. In brief, dataflow tasks were evaluated as sub-workflows and then micro-benchmarking has been conducted by running each sub-task in a single-core machine. Figure 3 shows a typical IoT dataflow with four types of IoT data processing tasks. ''Extract Data'' block involves stream pre-processing, data normalization and quality check. ''Predict Behaviours'' block includes applications for determining the future state of an IoT system based on a pre-trained models. ''Train Models'' involves building and training statistical models based on historical data. In ''Show Statistics'' block, related tasks like statistical aggregation and analytics   are applied over data streams to understand the behavior of an IoT system. more related to batch processing. Table 3 presents the main characteristics of the three hybrid workflow models: small, medium and large.
We extended CloudSim [61] simulator to reflect the adopted resource model as an edge cloud computing environment. The simulated scenario comprises two cloud data centers (CloudA and CloudB) and edge nodes that reflect the edge cloud resource model. For each data center, various VM configurations, including processing power and cost were applied. CloudA and CloudB VM configurations are based on configurations provided by AWS and Microsoft Azure, respectively. Table 4 provides the resource model setup for the experiments. At the edge layer, we simulated two edge resource setups, high and low capability. The former is used for resource estimation evaluation, while the latter is used for edge capability analysis in the context of hybrid workflow computation. Moreover, we consider the variation in networking quality and cost among the edge cloud system resources. Table 5 shows the bandwidth setup for the three type of resources.
Three experiments were conducted. The first experiment evaluates the performance of the proposed resource estimation technique against PSO-based and Full-mode techniques and with variation in estimation model parameters (arrival rate, window size and throughput). The second experiment measures scheduler adaptability and stability to maintain an optimized workflow execution in response to windows size and arrival reductions. The third experiment sought to identify the contribution of edge resource capability to optimization objectives (execution time T and cost C).

B. RESOURCE ESTIMATION EVALUATION
The evaluation of C-GDHW is based on its capability to optimize the execution of hybrid workflows in terms of execution time T and cost C concerning the variation of model parameters (stream window size ω, stream arrival rate λ, and execution throughput τ ) and workflow structure complexity (small, medium and large). The window size is the time interval of processing incoming streams (data messages) to generate the desired outcome. This involves applying functions for stream pre-processing, validation, aggregation, etc. The arrival rate is the number of received messages per unit of time. We defined the throughput as the minimum percentage of incoming messages to be processed during the window size. The throughput is a user-defined parameter which reflects the business requirements associated with the workflow execution. Maintaining a highly accurate predictive model is an example of high-throughput demanding applications.
To attain stable estimation results, simulation was carried out 30 times for each model parameter and workflow structure combination, and average values are recorded for each estimation technique. Model parameter values are varied in a range of 20% to 180% for both windows size and arrival rate, and in a range of 10% to 90% for throughput. Relative percentages for the default workflow configuration were proposed due to the variation in parameter values among workflow tasks. At 95% confidence level, results show an acceptable level of variation with reasonable data normality of sample size of 30. Table 6 shows a statistical analysis of large workflow execution time for each technique at the maximum percentage of each model parameter. The full-mode technique has almost no variance in results as it applies a deterministic model with minimal optimization. The PSO-based technique shows the highest variation in reflection to the randomization behaviour of the meta-heuristics technique. Next, each optimization case based on each parameter variation is discussed.    time with the increase in window size length. Even though all techniques are linearly responded to the change in window size (after 40%), the C-GDHW accomplished the smallest execution time increase, particularly for medium and large workflows with 45% and 60%, respectively compared to 60% and 75% execution time increase in the case of PSO-based for the same workflow scenarios. In addition, the proposed estimator was able to maintain the cost of adding more resources since more and more data streams are collected during extended data collection intervals. In comparison to the PSO-based technique, the C-GDHW performed 58% less cost for running the large workflow with 180% window size. This potential reduction implies the C-GDHW's ability to group a larger number of tasks and effectively perform periodic resource provisioning. This feature allows IoT applications to produce more data by adding more IoT devices or distributing incoming workload on current computation resources.    The results revealed C-GDHW capability to maintain a minimal increase in the execution time curve regardless of the stream speed. The PSO optimizer requires more execution time in response to change in stream speed, particularly with long-term execution workflow. Figure 7 shows that for a small workflow with 12 stream tasks, both estimators demonstrate relatively similar costsaving behaviour. Contrarily, with 36 stream tasks, for large workflow in Figure 9, the proportional difference in cost reduction is increased as the arrival rate is getting higher. Reduction percentage increased from 35% to 70%, in the case of 40% and 180% arrival rate, respectively. An additional    advantage of the proposed optimizer is the linearity behaviour in provisioning more resources (related to the monetary cost) to overcome high rate streams.

3) THROUGHPUT
In this work, we studied the contribution of application throughput on workflow execution time and cost. The proposed optimizer was capable of reducing the influence of high throughput, particularly for the monetary cost. For small workflows, as shown in Figure 10, the cost increases exponentially with the increase in throughput. Interestingly, the C-GDHW optimizer shows ability to control the cost (efficient resource utilization) as workflow model is getting more complex as depicted in Figures 11 and 12.
For execution time, the proposed optimizer linearly handles the increase in application throughput. Likewise, the optimizer produced potential cost saving compared to PSO of 46% with the highest throughput of 90%.  Furthermore, as the system progresses to high throughput, the C-GDHW and PSO are not sufficiently outperforming the full execution mode in reducing the end-to-end workflow execution time. This clearly stated the contribution of migrating more data to the cloud since edge nodes do not collaborate to handle data flooding from IoT devices. Moreover, the high workload at the network edge can degrade the performance of delay-sensitive applications [62]. Thus, a problem still to be solved is how to support the execution of delay-sensitive applications at the edge layer.

4) ADAPTABILITY ANALYSIS
We studied the C-GDHW optimizer behavior in reducing stream task arrival rate and window size to optimize workflow execution cost. Such reduction in optimization parameters has a direct impact on workflow execution outcomes. For example, in predictive analytics, the amount of data plays a vital role in training prediction models and produce highly accurate results. On the other hand, processing more data produces additional overhead on computation cost. Thus, it is essential to provide a balanced estimation to handle the trade-off between performance and cost. Figure 13 demonstrates the optimizer potential in providing a stable arrival rate and window size reduction for a large-scale hybrid workflow. For example, with variation in throughput, the optimizer provides the ability to preserve stable reduction for the arrival rate and linear reduction for the windows size in responding to the exponential cost increase. Figure 14 shows the relation between the capability of edge resources and workflow execution time and cost with the variation in stream arrival rate. Edge capability indicates the computation performance of edge resources. Table 4 provides details about low and high edge resource configuration, low capability refers to the small and medium types and high capability refers to large and xlarge VM types, respectively. Despite variation in stream arrival rate, results indicate a steady workflow execution time with low and high edge capability, with a 15% average difference and minimal contribution of the workflow complexity. On the other hand, the figure implies a linear increase in monetary cost, particularly for the large hybrid workflow scenario. The cost increases gradually from 56% (lowest arrival rate) to 8% (highest arrival rate). Overall, the graph shows that as the stream arrival rate increases, dependence on high-performance edge machines becomes costly compared to low-performance machines. This conclusion will motivate researchers to consider and move toward commodity edge computing.

6) SCALABILITY ANALYSIS
The increase in optimization time can have a significant impact on the scheduling process, particularly for large workflows. In a more complex scenario, the scheduling process could be repeated at short intervals, for example, to provide an accurate decision about model parameters, such as arrival rate and window size. Figure 15 shows the advantage of the proposed estimation algorithm, over our previous work using the PSO algorithm, in sustaining the optimization time with variation in workflow complexity and number of tasks. The optimization time increases by 45%, 55% and 60% for small, medium and large workflows, respectively. For the GDS-based optimizer, the increase in optimization time remains constant with 25% on average as the workflow structure is getting more complex, which reveal the scalability of the proposed scheduling. Moreover, The low optimization time of the GDS-based optimizer suggests that linear optimization models should be adopted to solve complex workflow estimation and scheduling problems.

VI. CONCLUSION
We presented a hybrid workflow scheduling framework for an edge cloud computing resource model. The framework includes two stages. In the first stage, we proposed a resource estimation algorithm based on a linear optimization approach, gradient descent search (GDS), while in the second stage, we proposed a cluster-based provisioning and scheduling technique for hybrid workflows in heterogeneous edge cloud resources. The aim was to achieve a multi-objective optimization of execution time and monetary cost under constraints of deadline and throughput. In comparison to the two nominated scheduling techniques, namely, PSO and Full-mode, the proposed technique performed better to control the execution of hybrid workflows by efficiently tuning several parameters, including stream arrival rate, processing throughput and workflow complexity. For large and complex workflows, the proposed scheduler provides significant improvement in terms of execution time and monetary cost in comparison to the PSO-based optimizer with an average of 8% and 35%, respectively. Moreover, the GDS-based technique shows a significant reduction in the optimization complexity compared to the PSO technique, with 50% on average.
For future work, we are planning to experiment with hybrid workflow provisioning and scheduling on cooperative edge cloud computing. We are also planning to extend the work to provide a convenient cost model from user and service provider perspectives.  He has been conducting research in the area of cloud computing, since 2008, and contributed to diverse aspects in the field, including multiclouds, energy-efficient cloud computing, and edge computing. He is also one of the original designers and developers of CloudSim, a widely used simulator of cloud environments. He has coauthored more than 70 papers, which attracted together 12,000 Google Scholar citations. His research interests include big data, the Internet of Things, internet computing, and their application. He is a fellow of the Higher Education Academy, U.K., and a Senior Member of the ACM.
KENAN M. MATAWIE received the B.Sc. (Hons.), M.Stat., and Ph.D. degrees from the University of New South Wales, Australia. In the last 20 years, he has held a range of faculty appointments at the University of New South Wales, the University of Technology Sydney, AMP Society, and the University of Newcastle, where he is currently a Senior Lecturer of statistics and data science with Western Sydney University. His research interests include the application of statistics, especially statistical modeling, advance data analysis, and developing statistical methods motivated by real-world problems. Most of his research was generated from application in computing, business, management, psychology, actuarial, medicine, geology, engineering, and information technology. These research contributions have been published and presented in more than 70 international journals and conferences. He is a member of many professional societies particularly and has been a Committee and Representative Member of the International Statistical Modelling Society, since 2005.
BAHMAN JAVADI (Senior Member, IEEE) is currently an Associate Professor in networking and cloud computing with Western Sydney University, Australia. He is the Co-Founder of the Failure Trace Archive, which serves as a Public Repository of failure traces and algorithms for distributed systems. Prior to this appointment, he was a Research Fellow with the University of Melbourne and a Postdoctoral Fellow with the INRIA Grenoble Rhone-Alpes, France. He has published more than 100 papers in high quality journals and international conferences and received numerous Best Paper Awards at IEEE/ACM conferences for his research papers. His research interests include cloud computing, edge computing, performance evaluation of large-scale distributed computing systems, and reliability and fault tolerance. He is a Senior Member of ACM and a Senior Fellow of the Higher Education Academy, U.K. He served as a program committee for many international conferences and workshops. He has presented many keynote and invited talks in several conferences and universities around the world. He has also guest edited many special issue journals. VOLUME 9, 2021