YAFS: A Simulator for IoT Scenarios in Fog Computing

Fog computing is a paradigm that extends the cloud to intermediate network devices with computational and storage capacities. This allows the execution of applications closer to edge devices and end-users by allocating services in those intermediate devices. The placement of those services has an influence on the performance of the fog architecture. We propose a fog computing simulator for analyzing the design and deployment of applications through customized and dynamical strategies. We model the relationships among deployed applications, network connections, and infrastructure characteristics through complex network theory, enabling the integration of topological measures in dynamic and customizable strategies, such as the placement of application modules, workload location, and path routing and scheduling of services. We present a comparative analysis of the efficiency and the convergence of results of our simulator with the most referenced one, iFogSim. To highlight the YAFS functionalities, we model three scenarios that, to the best of our knowledge, cannot be implemented with current fog simulators: dynamic allocation of new application modules, dynamic failures of network nodes, and user mobility along with the topology.


Introduction
Cisco coined the term "fog computing" as an extension of cloud computing, placing computer services closer to the users [1,2,3].Approximately speaking, some network devices, called fog nodes, perform computational tasks or data storage functions in the same way as cloud entities.This novel application placement has some advantages, such as the reduction of latency time, a lower network bandwidth utilisation, a reduction in the cloud costs, and an increase in the reliability and fault tolerance through the geographical distribution of devices.
Another related concept similar to fog computing is edge computing [4,5,6].
The small difference between them lies in the localization of fog nodes.In edge computing, the nodes are at the edge of the network, near the users.
In the context of the Internet of Things (IoT), the role of fog computing is to leverage functionalities such as on-demand scalability, real-time interaction, better security and privacy management, battery power savings, streamlining of communications, and rapid service delivery, among others [7,8,5].Fig. 1 shows the topology of a network with typical en-routing entities in comparison to network nodes with computational and storage capabilities in the fog computing model.The workload requests are generated from the endpoints / things layer and are routed to services deployed in upper-layer devices.These services can be allocated in several intermediate nodes or in the cloud.
The placement problem of software resources for fog computing or edge computing is an NP problem that consists of the selection of the optimal network entity to deploy a user application.There are some constraints and optimization factors that influence this problem, such as the user location, hardware and software features of the network entities, link characteristics (e.g., propagation, utilisation, and bandwidth), user requirements, application decomposition (e.g., containers, microservices, and serverless functions), QoS, energy, and cost, among others.These factors affect the dynamical evolution of the user movement, link failures, network congestion, and application popularity, among other aspects.One way to evaluate placement solutions is through simulation.Simulators are enabling tools for modelling, analysing and evaluating the diversity of policies and configurations.
In this paper, we present a discrete event simulator focused on, but not restricted to, fog environments called YAFS (Yet Another Fog Simulator).YAFS is designed to analyse the design of applications and incorporates strategies for placement, scheduling and routing.We compare the YAFS characteristics with II) Workload sources: each workload source represents the connection of a user or an IoT sensor or actuator that demands a service.Each source is associated with a network entity and generates requests according to a custom distribution.The workload sources can be created, changed or removed dynamically enabling the modelling of the user movements in an ecosystem.III) Placement, scheduling and routing custom algorithms: these algorithms are defined by the user.The placement algorithm is invoked in the initialisation and runs along with the execution according to a custom distribution.
The routing algorithm chooses the path that connects the transmitter and the receptor, and the scheduling algorithm chooses the application that runs the task.The scheduling and routing algorithms are defined in the same manner since the path selection depends on which application is chosen.The existence of the scheduling algorithm allows choosing between different modules in case of scaling policies.By default, the simulator includes both implementations: a static placement and the selection of the minimum path between two entities where the application is deployed.
IV) Custom processes: custom functions can be invoked at runtime to provide flexible implementations of real events such as the movement of the workload sources, generation of network failures, and specific data collection using third-applications such as Grafana [18] (an open platform for analytics and monitoring of computer infrastructures and services).This paper is organised as follows: Section 2 describes the state the art of the fog and cloud computing simulators.Section 3 includes the justification of the use of complex networks to model the infrastructure network.Section 4 describes the design and some details of the implementation of each component of our proposal.Section 5 includes three cases studies (involving allocation of modules, failure behaviour of devices, and movement of users) and presents a comparative study of results with iFogSim in terms of performance and convergence.

Related Work
This approach is focused in the design of fog computing simulators.There are different simulators for several types of distributed environments such as cloud, grid, and fog edge.We know of four specific simulators regarding the topic of fog computing: FogTorch [20], EmuFog [21], EdgeCloudSim [22] and iFogSim [9].We first analyse some common features of these simulators and then provide more specific features of all of them individually.
We classify the simulators considering the following criteria, which, from our point of view, are essential for realistic modelling of fog scenarios.I) The first criterion involves the structure of the topology.The topology allows us to represent the infrastructure of the network.EdgeCloudSim and IFogSim use a hierarchical structure.In contrast, FogTorch, EmuFog and YAFS use a graph structure.In addition, YAFS supports the definition of subgraphs within a topology.Subgraphs can be used to represent isolated regions or fog colonies [14].The generation of topologies is a complex and hard task due to the number of elements and connections.There are definitions of topologies such as BRITE and CAIDA topologies.These formats are supported by EmuFog and YAFS but not by the other simulators.II) The second criterion is related to the coding of the scenario.Existing fog simulators include an API where the characteristics of the scenario are defined; YAFS also supports the definition of the scenario through JSON-based files.The topology can also be defined using this syntax.III) The third criterion involves the characteristics of the results.
If complex scenarios are designed with customizable policies, it is necessary to record all the events of the simulator in files.Thus, users can perform complex analysis of these records to find specific indicators.EdgeCloudSim and YAFS record these data to allow post-simulation analysis.IV) The fourth criterion involves the capability to perform changes in the fog scenario during the simulation.The modelling of realistic scenarios must include changes in the different strategies.EdgeCloudSim supports changes representing the movement of users in the infrastructure.Only YAFS supports dynamic scenarios in the next strategies: placement, path routing, service orchestration and workload or user movement.V) Finally, the last criterion is the programming language.All other simulators are implemented in Java, but YAFS uses Python.
We summarize these five comparative criteria in Table 1.The nomenclature used in the Policy column is related to the specific policy used: A means allocation or placement; R -path routing, O -service orchestration, and Wworkload or user movement.Dynamic Policies indicates those simulators that can change their strategies dynamically during execution.
After a global analysis of the simulators, we also present a more specific description for each of them independently.
FogTorch [20]  alarm IoT application as a case study with three components: a fire manager (an actuator to extinguish the fire), a database system, and a machine learning engine.The IoT infrastructure was based on three fog nodes, two cloud entities and nine network links among them.
EmuFog [21] is a set of scripts to transform a set of initial configurations (network topology and placement criteria) into the input of the MaxiNet [23] simulator.It uses a graph representation to define the network topology.The authors implemented some functionalities to simplify the process of selection of fog nodes in regards to the topological features of the graph.Our simulator also implements this process.We delegate this type of computational processes in a complex network library to obtain topological features that the user can integrate into the topology.EmuFog application representation comes from Dockers2 , a container platform that encapsulates an application in a standalone package.The evaluation uses three graph types (the Albert and Barabasi model [24], and real-world topology from CAIDA [16] and from the BRITE tool [17]) for representing the network.The authors analysed the edge-nodes and the most suitable placements in the evaluation.From our point of view, however, the type of application used and the relationships among containers is not clear.
EdgeCloudSim [22] is simulator based on CloudSim [10], which is one of the most referenced simulators in the field of cloud computing.Sonmez et al.
introduced functionalities such as mobility models, network link models and edge server models to represent more realistic scenarios.Thus, new additional results were provided such as the LAN delay, number of failed tasks due to mobility and average number of mobile clients in a specific location.They presented a scenario with three configurations: one tier, two tiers, and two tiers with an edge orchestrator.The edge orchestrator entity controls the selection of the tier in each possible task execution.This simulator incorporates new functionalities relative to the original but is restricted in the taxonomy definition and how the mobility is defined.The type of results is also limited to the CloudSim version.
iFogSim [9] is a CloudSim extension that supports the management of edgenetwork entities and the evaluation of allocation policies.The infrastructure is defined by a set of entities: fog devices (or fog nodes), sensors, tuples (such as a network link) and actuators.The application is modelled as a directed graph with modules (representing computational resources), edges (a data dependency between application modules), and loops (defining a sequence of edges that should be monitored along the simulation to compute the response time. In the article, the authors present two placement strategies that we describe in detail in the evaluation section: cloud-only placement and edge-ward placement.They introduce the simulator with two case studies: a latency-sensitive online game (namely, the EGG Tractor Beam game) and intelligent surveil-lance through distributed camera networks.Based on the iFogSim simulator, we use the application model in our simulator, introducing new improvements in the API, and we compare our results using the first case study and the two placement strategies as explained in the article.

YAFS architecture
YAFS uses a generic library for the generation of discrete event simulation scenarios called Simpy3 .The Simpy library contains functions for the definition of processes (active components) and shared resources (such as network links and queues).It performs the execution of the simulation in three modes: as fast as possible, in real time, or manually stepping through events.It can also halt the simulation in case of lack of interaction or with a fixed step size.
Simpy is a robust and stable DES implementation that we use to implement functions to control the atomic processes behind a fog domain: the transmission of workloads among network links, the computation of processes in fog nodes, and others issues that we describe below.
YAFS is defined by six main classes: core, topology, selection, placement, population, and application.

Topology and Entity modelling
We represent the relationships among network entities of a fog computing scenario through a graph model where the nodes are network elements, routers, endpoints, fog nodes or similar elements and the vertices are the network links.
As mentioned, we can apply complex network theory to this model.We use the NetworkX [25] library with functionalities for manipulation, visualization and extensively tested implementations.This library can import CAIDA [16] and BRITE [17] topologies, and it supports graph formats such as JSON, GML, GEXF, Pickle, GraphML, and Pajek.
The mandatory attributes to define a fog node are an identifier (ID), the number of instructions performed per unit of time (IPT) and the memory capacity (RAM).Developers can include other, customized tags to define the topology entities.In Fig. 3, we include two JSON-based definitions of nodes: one with a range of power consumption and a coordinate value and one that contains only the mandatory attributes.YAFS supports a flexible definition of entities in the same scenario.Using customized attributes, we can represent logical relationships such as virtualization, containers, microservices, and serverless functions using nodes and vertices.
Finally, a simulation contains a unique topology class.As mentioned, this class is a graph-based representation where a determined number of applications and their corresponding policies are deployed.That is, each application has a unique policy of allocation of resources (placement), a policy of selection and orchestration of services (selection), and a variation of the workload (population).
Furthermore, we can deploy customized controls that dynamically interact with the application and the simulation variables (such as failure generation or to improve the computational capacity of a node).

Application model
The application model is the same as that of iFogSim [9] and is based on a distributed data flow (DDF) [26].An application is defined by modules that run services and messages (or dependencies) among modules.Thus, a DDF is represented with a directed acyclic graph where nodes are modules that perform one action on the incoming data and edges denote interoperability between modules.This application representation enables the partitioning and scaling of an application, which is useful for real program models such as microservices [27] and serverless [28] paradigms.
We adapted the application definition with regard to the iFogSim approach to complying with our design principles: independence of the results and ease of export and reuse functions.This fact is reflected in each definition phase of an application: modules, dependencies, messages and results.The mandatory attributes of a message are instructions and bytes.The instructions affect the service time, and the bytes affect the transmission time.
In YAFS, all types of modules are defined with the same methods.iFogSim authors use the term dependency to represent the relationship between modules, and these modules do not start the execution until they receive a message; instead, we use the term message.These messages can be used for other applications that have the same modules.The transfer of messages indicates how to transform a type of input message into another output message.In YAFS, all transfers are defined, including the generation of messages in sensors or the reception in actuators and the generation within modules (periodic messages).
The decision to transmit a message within a module is also implemented, with two methods: fractional selectivity and broadcasting.The latter allows message transmission to all replicated modules.Finally, in YAFS, the response time is obtained independently of the declaration of loops (an internal control of iFogSim for monitoring tasks of sequence of dependencies between modules of an application), i.e., in iFogSim, if a loop is not declared before the simulation, the execution times of a sequence of dependencies between modules cannot be measured.
To understand the differences between iFogSim and YAFS, we implement the application used in the first case study of iFogSim: the EGG Tractor Game (Fig. 4).The game consists of three modules: client, concentration calculator and coordinator, performing processing of the messages generated in the EGG sensor; some results are visualized in the Display actuator.The modules are defined in lines 9-11; modules that will be workload sources or simple sensors are defined as sources and sinks.They are necessary only to define the application.The messages are defined in lines 11-13.The following attributes are required: name, module source, module destination, instructions, and bytes.
Finally, the remaining lines define the transmissions.This is where we define how a message is transformed into another and how a message is sent between modules (through a distribution, a selection or a broadcast process).The placements of workloads (source entities) are defined in the population policy.
Our implementation includes additional types of applications that can be modelled and their interactions with the workload generators or users with regard to other simulators.Figure 5 shows three application types.Each application is set up by software modules represented with boxes.Application 0 presents a hierarchical structure where the messages (identified with M ij) trigger other messages.In the example, post-execution of M 01 triggers M 12 and M 13 .In application 1, we can observe a self-message and an interaction with the user from other module.In the last case, application 2, there is a broadcasting message (M b1 ) that reaches all S 1 -module deployments.Each S 1 -module returns a message that is addressed by S 0 -module and finally returns a response to a specific user.This last feature enables the return of responses to the initial claimant.
The supplementary material includes several examples of applications.In one of them, we define an application based on the structure of application 0 (Fig. 5) and the correspondent scenario (population policy, allocation policy   To illustrate these type of processes, we describe the population definition where we map workload generators in the entities of the infrastructure.This procedure requires three steps.First, we need to choose the type of message that is generated from the workload sources.The messages are defined in the Application class, and they are requested to perform the execution of application modules.Second, we have to define the temporal distribution.Finally, we have to associate how many of these generators we wish to have in the nodes. We have included an example of population criteria in Fig. 6).Lines 1 and 2 define two temporal distributions.The first one starts at 3000 time units, and from that point in time, it triggers an activation every 300 time units.
The second one triggers an activation every 10 time units.Line 4 generates an instance of a predefined extended class of Population.In application A, there are two types of modules: workload sources and workload sinks (similar to sensors and actuators).Thus, Lines 5 and 6, through JSON-based syntax, define the allocation of each sink module (in this case, we incorporate all sink modules, and we duplicate the number in the same entity), and the allocation of each workload source with a distribution and a type of message.
We can extend the Population class to model more complex scenarios (Lines 8-11).In this sense, the new instance (Line 9) of an extended Population operation follows the distribution defined in Line 1.This DES process starts generating workload sources at a certain time point, after which it is activated every 300 time units.In each activation, it generates a new workload source with the characteristics defined in Line 11, and it is assigned in each entity defined in the array (top20Devices).Sink modules are generated in the initialisation phase.A simplified version of this Evolution class is shown in Fig. 7).In these types of processes, there is a mandatory function called initial allocation and, optionally, a function called run that is invoked dynamically according to the distribution.Internally, functions defined in the Core class are used for our modelling, such as deploy source or deploy sink.Note that the sim variable is the instance of the simulation.We can control the topology and the rest of the DES processes, together with the simulation execution.This variable is created in the Core class.
The rest of the classes (selection, placement and customized processes) present a similar structure and behaviour.We omit additional examples to avoid redundancy.

Results
There are two types of events recorded (namely, task executions and network transmissions), but users can record specific metrics with customized DES processes.The results are stored in two CSV files.
When a node performs the work associated with a message, the simulator records the following attributes: id, type, app, module, message, DES.src,  DES.dst, TOPO.src,TOPO.dst, module.src,service, time in, time out, time emit, time reception.Specifically, id is an incremental integer value that remains constant during message propagation or transformation in other messages of the application.This approach allows controlling when an application partitioned in modules ends the execution of its complete service.The attribute type identifies the type of module (computational or sink).The attribute app identifies the application (the name attribute).module identifies the application module (the name attribute) that performs the service.message identifies the message (the name attribute).DES.src and DES.dst are the identifiers of the DES processes that send and receive the message, respectively.TOPO.src and TOPO.dst are the identifiers of the topology entities where the modules are deployed.module.srcidentifies the application module that sends the message.The service attribute can have a None value (if the message record comes from a workload source) or a numerical value corresponding to the service time.In a network transmission, YAFS records the following attributes: id, type, src, dst, app, latency, message, ctime, size, buffer.Here, id, type, app, and message take the same values as mentioned previously.src and dst are the identifiers of the topology entities that send and receive the messages.ctime is the simulation time when the action is performed.size is the size of the message.
Finally, buffer is an integer value that represents the number of messages in the whole network that are waiting for a link service.Consequently, a link can send only one message at a time, and messages have to wait for unused slots.
This value is an indicator of network saturation and is updated in each record.
In the case shown in Fig. 8, the transmission of the message from the workload source to the fog node generates three network transmission records.We show a sample of these types of records in Fig. 10.We have implemented some common methods (in the Stats class) to obtain more complex measures using the Pandas library [29].
Pandas is an open source library with several data analysis tools.To illustrate this data analysis, we include the next example in Fig. 11.The first approach follows the idea of sequences defined in iFogSim.From a sequence of messages (line 2), the showResults function provides the same results as iFogSim (line 4).In addition, we can perform more complex analysis.For example, we can compute the average latency each 300 units of time.

Evaluation
In the first section, we compare YAFS and iFogSim simulators in terms of performance and results using an application case defined in iFogSim [9].In the second section, we analyse the convergence of both simulators using the same experiments.
It is important to note that the results are not equal between both simulators, although we try to use similar settings.The definition of attributes is different in both simulators.These cases are the following: I) iFogSim uses the measure of MIPS in its computational devices.YAFS uses IPT (instructions per time.II) In iFogSim, the attributes of a link are included in the fog node using terms such as upBW and downBW, and there is another latency value in the connection between modules (i.e., eegSensor.setLatency(6.0)).In our case, the BW is defined in the link and has the same value in both directions.In addition, we define the propagation time, which is not included in iFogSim III) In iFogSim, a message has attributes such as tupleCPULength and tupleNwLength, corresponding to the number of millions of instructions and bytes, respectively.
In any case, temporal distributions are the same in the experiments, and we try to use similar values in the previously described attributes.

Comparison with iFogSim
We use the first case study presented in the iFogSim paper (namely, the EGG Tractor Beam game) for the comparison between both simulators.This application consists of 3 modules: client, concentration, and coordinator, and the experiment deploys the modules in a hierarchical three-based topology with a cloud entity that is linked to a gateway where all fog devices are connected.
The network can be scaled from the gateway device generating several subgroups.Fig. 12 represents an example of two topologies with 10 and 18 gateway subgroups.
We analyse two different placement strategies: a cloud-only placement (cloud policy) where all modules are deployed in the cloud entity and an edgeward placement (edge policy) where the modules are deployed in fog devices (orange nodes in Fig. 12).Both strategies are explained in the iFogSim paper.From the simulation, we analyse the following data: execution time and response time.
In addition, we vary the number of fog nodes: 4, 8, 12 and 16.The simulation is executed in a machine with 8 i7-cores running at 3.745 GHz with 8 GB RAM.
Because of the stable convergence of both simulators, as described in Section 4.2, we performed each experiment only once.Circle marks correspond to cloud policy and star marks to edge policy.At first glance, the behaviour of both simulators is quite similar, but we can appreciate some differences: I) In the cloud policy, a greater number of transmissions must be made since all messages go through more network links to compose the cloud entity.This volume of traffic generates a saturation in the network that affects the iFogSim runtime; II) edge policy generates more application modules; there are more DES processes to control each module, and this fact slightly affects the YAFS runtime.An increment of the simulation runtime is reasonable as more modules are controlled; however, the saturation of the simulated system not should affect the simulator itself.
The network is saturated with the parametrisation of the cloud policy experiment.The saturation is greater when there are more network devices and is proportional between different gateway subgroups.In Fig. 14, we represent the total number of messages waiting for the service in each level of fog nodes using YAFS.iFogSim does not provide this measure.messages in iFogSim before simulation execution.In YAFS, this step is not necessary; those latencies are calculated post-simulation.
With the edge policy, the clients and fog devices entities always have the same network distance, and the response time has to be constant and independent of the number of fog nodes.Table .2 shows that the convergence of the latency time is better in YAFS.Note that the seed of the random numbers is always the same in each increment of gateways in both simulators.This seed changes only with the number of messages and the simulation time.
With the cloud policy, the coordinators are allocated in the cloud entity.
Subsequently, more messages are transmitted across the network and must pass through the same link.In both cases, the latency presents an exponential trend (Table .2).The parametrisation avoids the network saturation in the iFogSim execution with 4 fog nodes.

Convergence
We analyse the convergence of YAFS using the same example of iFogSim as in Section 4.1.In this case, we use the edge policy since it is a stable configuration  3: Convergence of YAFS and iFogSim using 50 samples with an edge policy configuration and different number of fog nodes with 10,000 time units of simulation.

Three complex scenarios
In a second experimental phase, we highlight selected YAFS features, and we implement three dynamic IoT scenarios: allocation of new modules, failures on the infrastructure, and user mobility.
We analyse the behaviour of the results of these three experiments, but we have not been able to compare them with real cases.To the best of our knowledge, there are no public data from these types of scenarios, and the real characterization of some of these studies is outside the scope of this article.Thus, we have used arbitrary values (average values, distributions, etc.) to describe the expected behaviour of the results.
The first step is the definition of the network infrastructure (or topology).
To illustrate the use of complex networks, we use the Graph Stream Generator library [30] to create a Euclidean random graph [31].This topology is the same for all three experiments (with a size of 400 nodes and 2242 edges) where the links have the same propagation speed (1 time unit) and fog nodes can serve an unlimited number of modules.We choose this type of graph since such graphs represent social relationships among individuals and have a high connection degree.
The application consists of two modules: senders and receivers, and it has only one type of message.In this way, complex data analysis is avoided in the experiment.Initially, we randomly allocated 100 senders in the topology, and the number of receivers depends on each case study.Each sender generates a message each 10 time units, and the service time of the receiver is 0.0.Thus, the response time is equal to the latency time.To ensure accurate replication of the experiments, the seed of the random number generator is the same for all the experiments.In these three experiments, the results (latency times) are average values from the simple sequence between a sender and a receiver.The computation is similar to lines 6-8 from Fig. 11, i.e., the value is the average aggregation of a set of values of a time period.The selection policy is based on the minimal path distance between a sender and a receiver.
We allocate the receiver modules selecting the nodes with the biggest betweenness centrality of a graph [32].A higher value of node centrality corresponds to a greater occurrence of the node as part of the shortest path between two other nodes.The goodness-of-fit evaluation of this measure as an indicator to select a network device such as a fog node is not part of this study, but some analyses have been performed in previous studies [33].All three experiments and results are available in the code repository in the example folder5 .

Dynamic allocation of modules
In the first scenario, we scale the number of receiver modules.The objective is to observe how the latency time improves as this number grows.In the initialisation phase, 100 senders are deployed with one receiver, which is deployed on the node with the highest betweenness centrality.From time point 3000 of the simulation time, a new receiver is added with a period of 300 units.This process is repeated 19 times (a total number of 20 receivers are deployed).With this experimentation configuration, we observe that with 20 fog nodes, the response time tends to be stable.We can observe that the network is saturated with only one receiver (from 0 to 3000 in simulation time) because the latency is continuously increased.From the first deployment (time point 3300), the latency is reduced due to a higher number of available receivers, and the messages are more evenly distributed across the network.From the fifth module, however, the inclusion of new receivers along the network does not introduce any improvement since previous receivers still receive the workload.Note that the selection process of receivers is based on the minimum path between a source and a destination node.Starting at the eleventh module, the allocations have an impact in the selection strategy, and the latency is again stabilised.

Dynamic failures on network devices
In the second scenario, we implement a dynamic failure of nodes where the failure rate is based on an exponential distribution.The objective is to observe how the latency time worsens as this number of failures grows and consequently to show how the simulator can implement dynamic scenarios.In this experiment, we remove only the fog nodes and other network entities.Sender modules are not removed to ensure that the workload is the same throughout the simulation.
When a node fails, the node and its links are removed from the topology, which can affect the internal processes that the simulator handles.Thus, a new routing is computed for the messages that had a path through the failed node.If there is no other possible path, the simulator catches and records this outcome in a log.When a removed node has waiting messages to be served, the messages are discarded, and the simulator records this case.
In the initialisation of this experiment, there are 100 senders and 20 actuators deployed.All of them are allocated in the same nodes as in the previous experiment.The failures are generated from 500 time units and beyond with a mean of 100 time units.At the end of the experiment, the number of nodes available is 314, and the number of links is 1359, i.e., 86 nodes and 883 links  We represent the topology in Fig. 16a, where red coloured nodes are randomly chosen to be removed during the simulation.There are five red coloured nodes (fog nodes with allocated actuators), which will be removed.
In Fig. 16b, we represent the evolution of aggregate latency times (samples are aggregated each 100 time units), and the failures are represented with black lines or green arrows in the upper part of the graph.A black line marks the failure of a network device, and a green arrow represents the failure of a fog device.As we can observe, the latency worsens as failures occur.

Dynamic movement of message senders
In the third scenario, we wish to characterise the movement of users (sender modules) in the infrastructure.Therefore, receivers are statically located in the nodes, and the allocation of senders is changed in periodic steps.As a result, for every period, we reduce the hop count by one between senders and receivers.
The objective is to observe how the latency time improves and how the simulator can model scenarios with dynamic workloads.
In the initialisation phase, there are 100 senders randomly allocated and 20 receivers.All the receivers are allocated in the node with the highest betweenness centrality.Every 400 time units, all the senders are moved to the next nearest node with regard to the receiver nodes.We use the shortest path function to compute the next node.As the links around the receiver nodes receive many requests, we reduce the generation rate of the requests (100 time units).In addition, the selection policy of this experiment includes a round robin scheduler to select different receivers.
Figure 17a and Fig. 17b represent the topology (same layout configuration than previous experiments).The green coloured nodes represent the initial localisation of the senders (Fig. 17a), and at the end of simulation (Fig. 17b).
The pink coloured node contains the actuators.
The latency decreases at each step and ultimately converges at approximately 4.5 time units (Fig. 17c).The latency is obtained from the aggregation of the time series of events every 100 time units.Most of the senders pass from the node with most closeness to the actuators node in an average of 5 steps.
In summary, the customisation of temporal distributions and the structure of the YAFS engine enable a direct and flexible control of any type of event inside of the DES engine.Another notable aspect of the YAFS design is that it is based on a style of open programming, maximising the use of third-party libraries for delegating internal tasks such as the generation of topologies, visualization, or data analysis.For instance, as we show in experiments, we use complex network theory to perform several studies, and we export the topology to other graph formats for debugging and visualisation.

Conclusion and future work
We present a fog computing simulator for modelling novel and complex IoT domains.Our simulator, called YAFS, meets several design objectives: a light syntax, a user customised configuration of policies and a dynamic invocation of policies during the simulation, a definition of network topologies based on  complex network theory, and the capacity to record computational and transmissions results in a CSV format.This last point makes the simulator ideal for enhancing interoperability with third-party libraries, such as Grafana, for the creation of control panels to simulate monitoring infrastructures, or Panda or R for data analysis.In addition, the infrastructure and policies definitions can be done following a predefined JSON-based format, which simplifies use by non-expert programmers and facilitates the integration of results of optimisation algorithms for the evaluation of fog placement proposals.
YAFS is a simulator based on discrete events where the management of shared resources and simulation control relies on the Simpy library, which was specifically designed to simulate scenarios under Python.The infrastructure is modelled using complex graphs, which allows efficient delegation of the infrastructure management including topological features, that can be integrated into the customised policies.These policies are the definition of allocation modules in the fog nodes, the location of the workload generators (such as sensors or users), and the selection of resources to compute tasks (including both path routing and orchestration and the scheduling of jobs).The application model is based on a distributed data flow (DDF) defined in the iFogSim simulator.
YAFS is available in a code repository6 containing the implementation of all previous cases in addition to several examples and a documented API.
Regarding the evaluation, we compare two policies (cloud and edge allocations) with iFogSim.In both policies, the convergence of the results is similar.However, the YAFS runtime is slightly better than that of iFogSim.As YAFS has more functionalities, we design three complex experiments that are not compared with iFogSim since they cannot be implemented under its API: in the first experiment, we create new fog nodes; in the second one, we dynamically simulate failures of devices; and in the third experiment, we represent the movement of workloads in the infrastructure.The results are consistent with the expected values in each experiment.Future work will mainly cover the development of power-aware management policies, functions for controlling the computational capacity of the resources and improvements in the nomenclature.

V)
Post-simulation data analysis: YAFS performs automated CSV-based logging of two types of events: workload generation and computation, and link transmissions.The results are analysed post-simulation, which generates less overhead, avoids repeating the simulation to re-analyse other indicators, and enables the shareability of raw results.YAFS includes functions to obtain metrics such as network utilisation, response time, network delay, and waiting time, and other data analysis can be implemented by the user, such as the number of requests above QoS requirements or the peak of transmissions in a link.VI) JSON-based scenario definition: YAFS supports the importation of the scenario definition from JSON-format files.It enables the use of third-party tools that generate scenarios in a common JSON format; in addition, non-expert developers can use basic functionalities of the simulator.YAFS is developed in Python following the style guide PEP8 [19].It is available under MIT licence in a code repository 1 with detailed documentation, a tutorial and several examples.The contribution of this paper is the design of a highly customisable and adaptable simulator and the design of JSON-based files for analysing mobile IoT scenarios under the fog and edge computing paradigms.

Figure 2 Figure 2 :
Figure 2: The YAFS architecture is defined by six main classes: Topology, Core, Application, Selection, Placement, and Population.

Figure 3 :
Figure 3: Definition of two fog nodes using a JSON-based representation.

Figure 4 :
Figure 4: Definition of EGG Game application presented in iFogSim [9] using our API.

Figure 5 :
Figure 5: Three application types with a simple message passing, a loop message and a broadcast message.

Figure 6 :
Figure 6: Declaration of two population policies: one static (popA) and the other dynamic (popB).

Figure 7 :
Figure 7: Structure of a population class with three mandatory functions: init, initial allocation and run.

Figure 8
Figure 8 shows the four timestamps involved in the transmission of a message from the source to the destination entity where the software module performs the action.The label time emit is the value that represents the emission time of a message in a module source.The label time reception represents

Figure 8 :
Figure 8: Logged and computed times present in the fog node computation of a message.

Figure 9 :
Figure 9: Fog node computations recorded in a CSV file for subsequent analysis of the results.

Figure 10 :
Figure 10: Network link transmissions are recorded in another CSV file.

Figure 11 :
Figure 11: Analysis of the simulation results.

Figure 12 :
Figure 12: Network topologies with 10 fog devices (left figure) and 18 fog nodes (right figure).The large green node represents the cloud entity, pink nodes are the proxies, orange nodes are the gateways or fog devices, and small green nodes are client devices with one sensor (purple) and one actuator (blue).

Figure 13 shows
Figure13shows the execution time in both policies with regard to the increment of fog nodes.Blue lines are the results of iFogSim and green lines, YAFS.

Figure 13 :
Figure 13: Execution time comparison between cloud and edge policies with a different number of gateways: 4, 8, 12 and 16.

Figure 14 :
Figure 14: Number of messages enqueue (waiting) by network saturation using YAFS with cloud policy in increments of fog devices (4, 8, 12 and 16).

Figure
Figure 15a represents the network, where green nodes contain the senders and the size of the nodes represents the betweenness centrality.The results of the execution are shown in Fig. 15b.The blue line is the evaluation of latency (a) Euclidean random topology where the senders are allocated on green nodes and the size of the nodes represents the betweenness centrality.(b) Evolution of the latency time (blue line) relative to the number of allocated fog nodes (green dotted line).
(a) Network topology where red nodes are randomly chosen to be removed.(b) Evolution of the latency with failures on fog nodes (green arrows) and failures on network nodes (black lines).

Figure 16 :
Figure 16: Second scenario: dynamic failures on network devices.
(a) Initial localisation of sender modules (green nodes) on the topology.Receivers are in the pink node.(b) Final localisation of sender modules.(c) Evolution of the latency time during the simulation.

Figure 17 :
Figure 17: Third scenario: dynamic movement of workload sources.

Table 1 :
uses Monte Carlo simulations to determine the best allocation for an application through QoS indicators such as latency, bandwidth, cost, and Comparative table of fog simulators.
To obtain these values, we use the Pandas time series functionalities to sample the records in that time period and to apply the

Table 2 :
Latency time with two different policies: cloud and edge varying the number of fog nodes and the simulation time of the system.We run both simulators 50 times with a simulation time of 10,000