A Comparative Study of LoRa and IEEE 802.15.4-Based IoT Deployments Inside School Buildings

IoT deployments for smart cities and smart buildings have been multiplying exponentially in recent years, benefiting from a steady rise in the number of new technologies that deal with the underlying networking and application challenges in indoor and outdoor spaces. Due to the overlap in their specifications, we are still trying to figure out which of these technologies fits better to certain application domains, such as building monitoring. In this work, we provide a comparative study between IEEE 802.15.4 and LoRa, based on our experiences from using both wireless networking technologies in the context of indoor deployments aimed at IoT-enabled school buildings in Europe. We provide an apples-to-apples comparison between the two technologies, comparing them in some cases in the same building and application context. Although these two technologies initially might not seem to be competing in the same application space, in practice we found out that both have strengths and weaknesses in the specific application domain we have been using them. Moreover, our LoRa-based networking implementation, on top of Arduino-based hardware, appears to be an option that allows for a robust, reliable and lower overall cost IoT deployment, especially in cases with multi-floor building installations and low bandwidth requirements. We also present a network-level dataset produced from our installations and upon which we based our findings and discussion. We provide data collected from 6 different school buildings, 8 networks and 49 devices, to compare the performance and cost-effectiveness of competing IoT technologies. In that effect, with LoRa we can achieve similar or better link quality to IEEE 802.15.4, with higher data rate and lower costs.


I. INTRODUCTION
The Internet of Things (IoT) is quickly transforming the ways we work and live, accelerating the pace in which the real and the digital world interconnect. This is especially apparent in the domain of smart city applications, where we have been using a plethora of IoT-related sensing and networking technologies to implement systems for realizing novel application scenarios: monitoring the traffic in city streets, noise and air quality in city centers, energy consumption in our houses, integrating smart grid capabilities into power The associate editor coordinating the review of this manuscript and approving it for publication was Vyasa Sai. infrastructure, and so on and so forth. Since there is this huge differentiation between the application requirements in all of these domains, the industry and the scientific community through the years has responded by producing a plethora of wireless networking technologies to tackle all of these use cases. IEEE 802.15.4 [1] was one of the first technologies to come to the market, while technologies like LoRa [2] and NB-IoT [3] are more recent additions to our toolkit of IoT technologies, taking advantage of the work done in the field of 3/4G and aiming to cover additional spaces in the existing IoT and smart city market.
Furthermore, it is obvious that there is an ongoing paradigm shift in certain parts of this market, moving VOLUME 8, 2020 This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/ gradually from WPAN (Wireless Personal Area Network)/WLAN (Wireless Local Area Network) solutions to LPWAN-based (Low Power Wide Area Network) ones. Essentially, we have been shifting for some years now from the requirements coming from the initial era of wireless sensor networks, during which IEEE 802. 15.4 was first conceived (in 2003), to ones that are now becoming more common in the IoT era. More specifically, the requirements coming from novel IoT applications are pushing for long range communication capabilities; where in WPAN/WLAN communication range is in the order of a few to 100 meters, in LPWANs the goal changes to hundreds of meters or to a few kilometers in cities, and to tens of kilometers in rural areas. The second requirement is low power operation, with battery lifetimes for IoT devices ideally being several years long, while the third one is low cost, with the cost of the devices kept low in order to be able to implement large scale networks more easily. Such characteristics of LPWAN solutions are discussed in [4]. Moreover, in the case of LoRa, SigFox and NB-IoT there are additional dimensions to consider, with respect to the use of LPWAN solutions based on license-free radio frequency bands to build a more customized solution, or the adoption of the new 5G-based solutions and services from established telecommunication providers. The flexibility given in the first case allows developers to build an application tailored to very specific application requirements when needed, while in the second case a more standardized service could work better for use cases that need more reliability and guaranteed QoS. These technologies have taken different approaches: NB-IoT uses licensed bands and is offered by specific telecommunication providers in each country, SigFox aims for a more global solution through local partners enabling connection to the global SigFox network, while LoRa and LoRaWAN (the upper networking layer usually combined with LoRa) allow developers to build private networks, like in the application that we discuss in this work.
In this developing environment, one of the most interesting application areas is related to sustainability, with a focus on energy consumption. In the European Union, a lot of research activity has focused on the use of energy in public buildings and initiatives like Build Up [5] offer valuable sources of information in the area. An additional dimension to consider is that educational buildings constitute 17% of the EU non-residential building stock (in m 2 ) [6], while the educational community is one of the most interesting end-user groups to work with. Having the above in mind, we developed in the Green Awareness In Action (GAIA) project [7] an IoT platform that utilizes various IoT elements to produce real-world sensing data, online management tools and gamification to address the need for sustainability awareness and energy savings inside schools. Based on real-world sensor data produced inside a fleet of school buildings, educators and students had reliable data to implement policies for energy savings inside their school buildings, focusing on behavior change in terms of energy efficiency rather than retrofitting buildings with things like insulation, etc. At its current state, the GAIA infrastructure is operational in 25 school buildings spread in Greece, Italy and Sweden.
Because the project's progress from design to implementation and testing with real users lasted several years and went through a number of cycles (February 2016 to May 2019) allowing revisions to hardware and software design, as well as a number of conditions like limited availability of certain networking components and arrival at the market of new ones, in practice this meant that gradually shifted from using one technology to another. At the early stages of implementing this IoT deployment, we opted to use IEEE 802.15.4-based 2.4GHz modules, and more specifically the popular XBee family of such transceiver modules. However, although our results were satisfactory at an early stage of the project, as we added new school buildings to the project's infrastructure we encountered new requirements and restrictions, e.g., due to the buildings' characteristics, that made deployment more complex and pricier. There was also a lack of replacement for the initial modules that we used. For such reasons, and due to the fact that LoRa popularity is on the rise, we opted to try using LoRa modules instead for a number of schools. On the one hand, LoRa is by design positioned well for applications where the IoT devices basically upload data to a cloud server/network gateway like in our use case, versus utilizing downlink communication. On the other hand, IEEE 802.15.4 and other similar technologies are overall better suited than LoRa for use-cases with more symmetric bandwidth requirements. Also, the frequencies utilized in LoRa have an advantage compared to the 2.4GHz band that is more frequently used in IEEE 802.15.4, because they can penetrate through building walls and floors much easier due to their wavelength; IEEE 802.15.4 modules are available in similar frequencies, but in most cases the 2.4GHz bands are used, or the transceivers are significantly pricier than LoRa ones. During the initial stages of setting up our LoRa deployments, we were not sure whether they would outperform or equal the performance of the IEEE 802.15.4-based ones, especially given the inherent bandwidth limitations of LoRa and the time-on-air restrictions imposed by regulations (e.g., ETSI regulations in Europe, cf. Section III.B), or the current scarcity of LoRa-based solutions in this specific application domain; however, that is not the case, as we have seen in practice through our experiences utilizing both technologies in real-world IoT deployments in public buildings.
In this work, we present a thorough comparison between LoRa and IEEE 802.15.4 as the backbone of an IoT deployment inside school buildings. Our aim is to present our practical experiences and lessons learned by trying to balance our designs for well-functioning and reliable infrastructure to build a set of certain applications over them [8]. We briefly discuss the two technologies and underlying technological assumptions and we discuss our rationale in their utilization in our use case, along with an analytic discussion on the effect and interplay between various application parameters like network topology design, layout of the IoT nodes inside a building and the distance between them, the choice of a specific Spreading Factor in LoRa, among others.
Moreover, through an apples-to-apples comparison conducted via experiments and measurements using both technologies at the same location and same time period, we evaluate how they perform within the same application settings. The experiments included in this paper demonstrate that, for our specific application and under the respective design constraints, LoRa can be more reliable in terms of actual data delivery rate, and it can also be adequate in terms of data rate, delivering basically the same volume of sensing data as IEEE 802.15.4 but at a lower overall cost, due to requiring a less complex network topology. In this sense, we showcase that LoRa can be a good alternative to other existing approaches for implementing applications that are not typical LPWAN use cases, i.e., with a rather small number of nodes inside a building, when configured suitably. We also examine the side effects of having multiple LoRa-based installations in close proximity to each other on the network level, while relaying our practical solutions to the issues that we faced. With respect to our contribution, we showcase that for application settings similar to ours, using different network frequencies might be preferable to using different LoRa spreading factors due to the effect of network interference. In addition, our results were achieved in real-world settings, over a long period of time, while our considerations were formulated upon practical experience from deployments which in some cases have been operational over several years.
Regarding our contributions in this work in the context of several network layers, for our IEEE 802.15.4 networks we use an XBee module with carrier-sense multiple access with our collision avoidance (CSMA/CA) protocol implemented as part of the MAC layer, as well as Cycle Redundancy Check (CRC) and a re-transmission mechanism. We implemented on top of the 802.15.4 MAC layer a custom tree-routing protocol, in order to realize multi-hop connectivity in large buildings. With respect to LoRa network, we worked on top of the physical LoRa module stack, implementing a custom network protocol, in which gateways request data periodically from each node in the network. The gateways also utilize a custom re-transmission mechanism for their requests, as well as employing CRC checks on the received packets.
Overall, in the LoRa networks we deployed we achieved very stable and reliable communication links, with no degradation due to the distance from the network sink all over the school building. In addition, we saved a 46.17% over the total installation cost in comparison to the equivalent IEEE 802.15.4 network. On the other hand, the equivalent IEEE 802.15.4 network for the same building showed variations in the communication quality between nodes of up to 8% with the advantage of offering a 3% better delivery ratio but suffered from degradation due to distance from the network gateway. Furthermore, our LoRa networks have, for the same period of time, achieved greater data rates with at least twice the amount of messages as evident by the dataset that we will present in the next sections.
It is important to note here that we have based our implementation on LoRa (i.e., the lower physical layer) and not LoRaWAN (i.e., the upper networking layer stack usually utilized together with LoRa). In our case, we decided to utilize the features offered by LoRa to build a solution tailored to the specifics of an IoT deployment aimed at public building monitoring. LoRa enables a long-range communication link that can be used for creating a point-to-point and star topology network inside buildings with bandwidth requirements that can be used on constrained devices, such as Arduino. However, we think that this use case scenario, and its associated application requirements, is generic enough to be of interest to a large part of the community interested in evaluating and using such technologies in similar settings.
Furthermore, we discuss here the dataset we utilized in our analysis, which was produced by our IoT deployment inside GAIA's buildings. This dataset contains data gathered from 6 different school buildings in Greece, 8 distinct networks and 49 individual devices, covering a time period of 83 days and has a total volume of 41 megabytes. We believe that it could help the community in understanding the parameters studied in this work, and potentially serve as input to other similar studies as well. We also provide a discussion on more practical findings and aspects of IoT deployments inside public buildings, in order to provide a more complete picture of our work.
Regarding the structure of this work, we continue by presenting previous related work on Section 2, and briefly discuss some key aspects of the two technologies studied here in the context of our setup in Section 3. Section 4 provides some further details on the actual IoT installations used to provide the data we worked with, while Section 5 proceeds with presenting an overview of the dataset we used. Sections 6-8 include the presentation and discussion of our findings from comparing the 2 technologies, and Section 9 provides a brief discussion comparing LoRa and IEEE 802.15.4 from the point of view of installation costs. We conclude this work in Section 10.

II. PREVIOUS RELATED WORK
As mentioned in the introduction, in recent years we have seen a multitude of new networking technologies competing in the smart city and IoT application domain. Since the research community has been very eager to test and use these technologies in practice, there is a lot of interest in works evaluating their strengths and weaknesses, especially when comparing them in the context of actual deployments. A recent work reviewing the real-world applications of LoRaWAN is [9], presenting a plethora of application domains where LoRaWAN has been applied to, technological aspects of LoRa and LoRaWAN, as well as important related trends in research in recent years. Reference [10] recently surveyed the various technologies available for LPWANs, arguing that it can be hard finding the right solution for a specific application VOLUME 8, 2020 and that also currently there is no one-solution-fits-all technology available. Reference [4] described the issues related to the LPWAN segment of IoT, comparing the state-of-theart in 2016 with the new contenders in the respective space in the field, like LoRaWAN. A recent study [11] regarding the long-term application of LPWAN technologies, and more specifically LoRaWAN in an actual air quality smart city monitoring setting, concluded that it is indeed a viable option for implementing city-scale applications. A similar study in [12] reached the same conclusion, suggesting that LoRaWAN can be utilized for implementing a number of smart city applications. Regarding the use of LPWAN in industrial application settings, [13] performs an assessment of the use of LoRaWAN in such settings, while also making a comparison with non-beacon based IEEE 802.15.4. Their conclusions reflect our own experiences described in this work, i.e., they consider it to be a viable alternative and easyto-deploy solution in industrial application settings.
Regarding recent comparison studies between protocols used for LPWANs in IoT, [14] and [15] discussed aspects related to LoRa, SigFox and NB-IoT. Both studies concluded that LoRa and SigFox possess very good characteristics in terms of capacity, battery lifetime and cost, while NB-IoT has an edge in terms of latency, reliability and quality of service. Reference [16] argued further that LoRaWAN possesses certain advantages due to its open architecture and the ability to implement private deployments, i.e., without reliance to third parties for their operation. Such technologies are all currently being used in smart city applications, while there are other recent technologies, like Sure-Fi [17], that try to fit in the context of use cases with more narrow application requirements. Another interesting technology gaining traction is IQRF [18], sharing several characteristics with LoRa. However, Sure-Fi is focused on long-range applications with very limited data rate requirements and has targeted the North America region until now, while the choice in IQRF hardware is currently very limited, possibly due to low market adoption, and its cost is higher than LoRa. Currently, there is a lot of interest in understanding the parameters related to their performance in the real world, and which one of all these technologies can handle better the respective use-cases. Such aspects are discussed in [19], where a smart city deployment using LoRa and IEEE 802.15.4 is evaluated using mostly simulation methods and limited real-world studies. Their results showed that for deploying a LoRa-based solution at a city scale, a careful tuning of parameters such as the choice of LoRa Spreading Factors (cf. Section III) and the use of multiple gateways is required for satisfactory performance. Such results agree in general with our own experience, i.e., in this work we describe how we had to fine-tune our own LoRa network deployments to reach a satisfactory performance level for our use case. However, in Section VII we describe how we used a different solution, i.e., use different frequency channels for each network, instead of different spreading factors, due to observing noticeable interference. Reference [20] also studied the limits of LoRaWAN as a LPWAN technology, concluding that smart-city, metering and tracking applications were good candidates for using LoRaWAN, but considered real-time monitoring not a good fit unless small-scale networks are used. In our experience, going with a custom protocol over LoRa without LoRaWAN has allowed us to address our use case within its requirements very effectively.
Although most related work discussing aspects like the ones mentioned above is limited to simulation, several recent papers feature data from real-world settings. LoRa performance is explored to a certain degree in [21], with a discussion on possibilities and limitations in environments like forest and cities, commenting that in urban environments range can be severely impacted, especially when gateways are placed at a low altitude. Reference [22] explores its scalability in the context of large-scale sensor networks, commenting that in the usual scenario where the SF with the lowest data rate and largest range is used, collisions deteriorate performance very quickly. Reference [23] presents an evaluation of LoRaWAN using a permanent outdoor deployment, focusing on its adaptive data rate mechanism; results showed that packet delivery ratio can be worse in some cases using this mechanisms, due to oscillations between different SFs in the nodes. Reference [24] provides an experimental study on LPWANs for mobile IoT applications, arguing that mobility impacts LoRaWANs performance significantly even in very low speed scenarios. Reference [25] provided a study of LoRa in long-range use-cases and produced certain radio propagation models to be used when designing LoRa-based solutions. Their work confirmed coverage of up to 8Km in urban and 45Km in rural areas (with line-of-sight). Reference [26] examines a use case scenario with several similarities to own work, using the RFM69 wireless radios, which utilize wireless technologies quite similar to LoRa and IEEE 802.15.4, concluding that this type of radio can present a viable alternative for this application domain. In comparison, in this work we cover in greater detail several issues related to the wireless networking technologies utilized in such use cases, aiming to provide insights related to the design and implementation of actual IoT deployments.
Overall, so far, most previous related works are either based heavily on simulation, or they do not attempt a straight apples-to-apples comparison between different networking technologies in specific use-cases e.g., for IoT, pervasive computing or smart cities. Our work here contributes to the discussion over which technology is better suited for real-world application in a representative use-case; school buildings are a characteristic and ubiquitous example of public building. Our application requirements in terms of data sampling and quality of service (QoS) are also similar to other related application scenarios (e.g., office building monitoring and automation). Moreover, there is a limited number of works that examine real-world settings, such as the co-existence of LPWANs and WPANs in the same location. Furthermore, while there are a number of works presenting open source power meters similar to the ones used in GAIA, they do not focus much on wireless communication and protocol aspects.
An additional important aspect is the need and benefits of apple-to-apple testing in wireless communication, especially since there is a plethora of papers, algorithms and solutions for various aspects of the wireless communication technologies that have surfaced in recent years. To a certain extent, this is also related to the issue of replicability of experiments in many scientific disciplines. In previous years, research projects like WISEBED [27] provided researchers with wireless sensor network testbeds to evaluate their solutions, e.g., MAC protocols, over the same infrastructure. The FIT IoT-LAB [28] is another, more recent, similar IoT laboratory that also offers LoRa and IEEE 802. 15 [32]. In such contexts, the organizers of such events try to provide the same environmental conditions (e.g., same buildings) for all competing solutions as a means to guarantee a fair comparison between them. In this work, in several experiments we utilized installations of IEEE 802.15.4 and LoRa inside the same building, at the exact same locations and within the same application scenario, in order to provide such a fair comparison, given the constraints and requirements of our use-case scenario.
With respect to more network-level aspects of LoRa and LoRaWAN deployments, [33]- [36] study aspects related specifically to LoRa and the various spreading factor options available to developers. [33] argued that inter-SF collisions are an issue contrary to the belief that SFs in LoRa are orthogonal, while [34] focused on chirp-spread spectrum, further showing that symbols are not orthogonal. Reference [35] studied the effect of co-technology interference in LoRa in order to study the potential by implementing multi-hop relay networks; their results show that such networks can handle time-synchronized transmissions and implement such mechanisms. Reference [36] studied theoretically the achievable uplink LoRa throughout, together with a practical evaluation and a showcase of LoRa's imperfect SF orthogonality. Reference [37] discusses issues related to inter-network interference due to multiple LoRa networks being deployed in the same location. We also study such issues in our work, providing some practical insights on how to deal with such conditions in real-world applications. However, we use different mechanisms to cope with such conditions, since we base our approach on using different spreading factors and not just multiple base stations or directional antennas. We also back our work through real-world experiments and not results derived via simulation, a method which we believe is more valuable for these cases. Reference [38] provides a good overview on the effects of the current EU regulations on duty cycle in long-range sub-GHz technologies like LoRaWAN, as well as discussing common misconceptions regarding these aspects and current research directions to solve the existing issues.
Regarding radio interference, WiFi (IEEE 802.11) and IEEE 802.15.4 networks more often than not share the same unlicensed 2.4GHz Industrial Scientific Medical band, although the latter is also available in sub-GHz bands. These bands suffer from heavy interference due to the ever increasing number of WiFi-enabled devices, resulting in interference that is much stronger for IEEE 805.15.4, due to its lower transmission power. Due to this fact, a lot of attention has been devoted to the evaluation of the performance of IEEE 802.15.4 in the presence of WiFi interference, even in the context of competitions targeting to minimize the effects of external interference [39]. Reference [40] presents an evaluation of the interference between both technologies providing a simulated comparison in the OPNET computer network simulator, while [41] uses the QualNet Simulator to compute the Packet Delivery Rate (PDR) statistics for ZigBee and WiFi networks. Based on its results, the effect of WiFi on ZigBee packets is highly dependent on the traffic intensity and number of nodes in each network. In addition, [42] presents an in-depth analysis of the effect of multiple 2.4GHz networks in different scenarios and in relation with the devices' positions. Experiments on real-world testbeds have also been performed like the ones at the ''European Laboratory of Wireless Communications for the Future Internet'' (EuWin) [43]. They report an extensive measurement campaign to measure the performance of a ZigBee network, when affected by different levels of interference and, when different types of traffic are generated. Moreover, [44] provides some real-world observations on the cross-interference between WiFi and ZigBee in indoor environments with mission-critical transmissions.
With respect to LoRa and interference, the chirp spread-spectrum modulation used in LoRa is based on chirps modulated with different spreading factors (SFs, cf. Section III.B) that are quasi-orthogonal [33]. The fact that SFs are not perfectly orthogonal among themselves produces inter-SF interference due to inter-SF collisions [34]. Reference [35], [36] studied through computer simulations and/or experiments, how such interference decreases LoRa's performance, especially for high SFs. Reference [45] studied the impact on the deployment of Internet of Things devices with a focus in LoRa and SigFox technologies in the 863 − 870MHz band. The study focused in signal activity and power levels measurement in diverse real time scenarios. The results presented showed that there is a 22-33% probability of interfering signals above −105dBm within the specific band in a shopping area and a business park in downtown Aalborg, which thus limits the potential coverage and capacity of LoRa in such locations. Regarding the interference between LoRa and IEEE 802.15.4 networks, [46] shows that LoRa can obtain high packet reception rates, even in the presence of strong IEEE 802.15.4 interference, while IEEE 802.15.4 also has some resilience to LoRa interference. VOLUME 8, 2020 Furthermore, inter-network interference due to multiple LoRa networks deployments in the same physical space have been studied at [37], showing that such interference can drastically reduce the performance of a LoRa network. Their study focuses on the interference by 4 closely located networks of 600 nodes each, and a transmission rate of 20 bytes by each node every 16.7 minutes. Based on that, the average network load is 2KB per minute. In contrast, our work focuses on study the inter-interference cases in LoRa networks in school buildings. Our use-case consist on 3 closely located networks with a total of 22 nodes, with every node communicating with the highest possible rate. The average network load we achieve is approximately 11KB per minute. The studies mentioned above focus on LPWAN applications with a big number of nodes utilizing very low data rates. In this paper, we study an application scenario that uses LoRa in a quite different application setting with a small number of nodes producing a relatively high number of data, and where IEEE 802.15.4 is considered a more ''mainstream'' approach and LoRa has not been considered as a contender until now.
Regarding our own previous work, the design and implementation of the cloud-based aspects of the project are discussed in detail in [47], while its open source hardware aspects are discussed in [48]. In [49], we presented some preliminary results related to the aspects presented in this work. Having presented this initial set of results, we present here a greatly evolved version of that work, crucially providing a straight comparison between the two technologies in the same location simultaneously, as well as discussing the dataset produced by our IoT installations that is utilized in this work and offering it publicly to the community. We also investigate in much greater detail aspects such as the effect of having multiple LoRa IoT deployments with close proximity and within different public buildings.

III. BRIEF OVERVIEW OF OUR IEEE 802.15.4 AND LoRa SETUP
In this section, we compare the two networking technologies and how we utilize them in practice. On the surface, there are many differences between them, and most would argue that they are fit for completely different scenarios. LoRa is the newer protocol, targeting long-range use-cases; in contrast, IEEE 802.15.4 was designed to serve more conventional use cases, with a communication range of up to 50m. However, in practice we see that the two technologies are increasingly utilized in similar situations, and this is the reason why we compare them here: we wish to compare their strengths and weaknesses for an indoor use-case scenario, in buildings where potentially there is an IoT network covering multiple floors and rooms. Table 3 provides an overview of the characteristics of the two transceiver modules that we used in our study.
A. IEEE 802. 15.4 For our IEEE 802.15.4 network, we chose to use the S1 and S1 Pro Digi XBee network modules [50]. We use the S1 Pro module when we need extended communication range inside buildings. In the rest of the text, we use interchangeably XBee and IEEE 802.15.4. We configure every XBee module to use a IEEE 802.15.4 MAC mode with acknowledgments (ACK). The RF module operates in a unicast mode that supports retries. The receiving modules send back an ACK for RF packets to confirm each reception. If a transmitter does not receive back an acknowledgment, it will resend the packet up to 3 times, until the ACK packet is received, or drop it after four trials in total. The transceivers are operating within an ad hoc network topology with no master/slave relationship and each node can have both roles. The Network ID and Channel are identical across all the modules in the network, while each RF packet can have a maximum payload of 100 characters, i.e., its maximum payload size is 100 bytes.
We used these specific modules and not e.g., XBee-PRO at 868MHz due to limited availability and higher cost of these modules. In fact, one of the reasons that we decided to experiment with LoRa was this limited availability of some of the original XBee series modules that we used and the resulting difficulty in replacing malfunctioning IoT nodes in our deployments. Another aspect that should be taken into account is that 2.4GHz 802.15.4 modules at the moment seem to be more popular than sub-GHz ones. In this case, had we used sub-GHz modules we would have similar bandwidth restrictions to LoRa due to this frequency band. Overall, we think the comparison between the two technologies is fair, in the sense that we used them in a real-world setting, at the same locations, under the same constraints, while also having performed tests to ensure that each solution was deliverable acceptable results in the context of our application.

B. LoRa
Regarding the features of LoRa that we examine in this work, it uses an MFSK modulation together with Chirp Spread Spectrum (CSS). Each bit is spread by a chipping factor, with the number of chips per bit called the Spreading Factor (SF). Such chirps are used to encode data for transmission, while the receiver uses inverse chirps to decode the messages. Spreading Factors essentially define how the data transfer rate relates to transmission range, by indicating how many chirps are used per second, and represent the number of symbols sent per bit of information. The possible values of SF range between 6 and 12. In practice, SF dictates the data rate, e.g., SF9 is 4 times slower than SF7. To put it in other words, the slower the bit rate, the higher the energy per bit sent and, thus, the higher the range [51].
ETSI [52] regulates the 865-870 MHz band in Europe. We specifically use band number 48, used for non-specific short range devices on the frequency range 868.0-868.6 MHz. In Europe, for this specific band, there are 2 basic limitations: a) the maximum transmission power allowed is 25 mW (equivalent to 14 dBm), and b) there is a hard limit on the duty cycle set to 1%. The duty cycle is defined as the ratio, expressed as a percentage, of the maximum transmitter ''on'' time monitored over one hour, relative to a one hour period. To put it in other words, a LoRa device can occupy the medium only for 36 seconds for every hour [53]. Another relevant important aspect in LoRa is Time on Air (ToA), which is the time the medium is occupied by a node transmitting a message until it is received by another node in the network, and is directly related to the duty cycle. We can use ToA estimations to check whether we satisfy the duty cycle restrictions in the current regulations. To simplify things, LoRa utilizes ALOHA at the MAC layer, sending data whenever available. Naturally, this means that collisions increase as the transmission rate or network node density goes up. The regulations were extended in 2018 to include the use of the 47b band (865 -870 MHz) for data networks with non-specific short-range devices. In this case, the duty cycle is limited to 10% for network access points and 2.5% otherwise, while the bandwidth is restricted to 200kHz and requires Adaptive Power Control. In order to avoid the implementation of a complex and heavy Adaptive Power Control mechanism on our resource-constrained nodes, and with the aim to compare the LoRa capabilities for a diverse configuration with various bandwidth options, we use our modules at the basic 48 ISM band.

IV. A LARGE-SCALE IoT INFRASTRUCTURE INSIDE SCHOOL BUILDINGS
In this section, we proceed with the description of GAIA's infrastructure and its design goals; this will allow for a better understanding of how and why the networks used for our comparative study are set up and used. As mentioned in previous sections, GAIA's goals were to increase sustainability awareness in students and educators, as well as implement energy savings in several schools through behavioral change of these users, via a series of educational activities that were based on real-world data. These data were produced by the IoT infrastructure that we installed in the school buildings that participated in the project, aiming to provide the educational community with real-time, real-world data from their surroundings. In this context, we opted to go for energy consumption data through power meters installed at central points in the buildings (usually the main electricity distribution panels) using the SCT-013-030 sensor, as well as environmental data inside a number of classrooms inside each school by a set of sensors (light, noise, temperature, humidity, activity, particulate matter, etc.) using the THO2 (temperature/humidity), TSL2561 (luminosity), LM386 (noise levels) sensors and a simple 5V PIR module. More details on the exact build of our sensing devices can be found in [48]. Figures 1 and 2 showcase some examples of the nodes' placement inside classrooms.
Additionally, the IoT infrastructure had to be inexpensive as it was going to be installed in a large number of buildings with a small budget. It also had to be installed in ''hostile'' environments, with young and older students, that are always curious about their environment and additional damage on some of the devices was to be expected except for the normal hardware failures. We also decided to base our work on the Arduino platform, 1 an open-source hardware and software  platform that is extremely popular with educators and students of all ages. Using Arduino, the students were able to better understand what the devices installed in their school are doing and, in some cases, even help assemble and program some of them as in school activities and workshops. However, limited memory resources in Arduino nodes constrained us, to a certain extent, in terms of the features included in our implementation.
The results reported in this article are based on the outcomes of the GAIA project that focused only on the IEEE 802.15.4 network topologies presented in the next section. As a result, we were not able to interfere or modify any of the existing physical installations to modify the placement or the density of the devices in each school. Additionally, GAIA had two major requirements from the IoT infrastructures to be deployed. The installations needed to be as maintenance-free as possible as the school personnel was not going to be involved in its maintenance and frequent travel to the installations was also not feasible. We also needed to have as fast sampling as possible using the hardware that was decided before the kick-off of the project. As a result, all devices had to be powered by wall adapters, to minimize the need to maintain batteries, use solar charging or other technologies for energy harvesting and keep the devices operating at all times reporting data as fast as possible (around 6 messages per minute in the IEEE 802.15.4 and from 6 to 12 messages per minute for LoRa).
A. IEEE 802.15.4 NETWORK TOPOLOGY To implement our IEEE 802.15.4 networks, we utilized XBee modules for each IoT node, for which we used the popular Arduino XBee [54] and XBeeRadio [55] software libraries. The IoT nodes form adhoc networks and report their measurements to the cloud infrastructure through one or more IoT gateways for each school. As IEEE 802.15.4 has a relatively short transmission range (usually up to 50m in indoor open spaces), in most cases end-to-end communication is not possible across large buildings, due to propagation obstacles (e.g., thick concrete walls, as in the school buildings in Greece). We use our own routing implementation, as we operate on top of the pure IEEE 802.15.4 network and not on top of full protocol stack like ZigBee [56]. In our custom implementation, all indoor IoT nodes form an ad hoc overlaying multi-hop bidirectional tree network. The gateway is the root of this tree topol of the network (Figure 3). This is also partially related to the current system being a continuation from some previous wireless sensor network projects in which our team participated in, like Wiselib [57] and Spitfire [58].
We use a simple strategy for the nodes wishing to join our network. Gateways broadcast ''heartbeat'' messages periodically that are propagated through the established edges of the network. If a node that e.g., has just reset or was activated for the first time and has not yet joined the network, receives such a heartbeat message, it becomes a child in the already established routing tree of the node it has heard the message from and if it has received this message with an RSSI lower than a threshold of -90dB. When we installed our tree-routing network implementation in a real school building, we noticed that the tree was continuously recalculated to find the optimal path to the root provided using the best and closest neighbor, in terms of RSSI. In the given school building, in which the doors and windows close and open frequently and a lot of people move inside it, the signal strength is very affected. This had an impact in the network performance that we tried to mitigate by creating a more stable tree structure by imposing RSSI thresholds. We introduced a threshold limit of −90 dB that we experimentally evaluated to guarantee a more stable behaviour, prioritizing the shortest path in terms of distance to the root device and better RSSI values. The routing library we developed for the Arduino and IEEE 802.15.4 devices and we use in our deployments is also available publicly on GitHub. 2 After the routing tree is established, each node collects environmental or other sensor data and sends a data packet (e.g., an Environmental Data Packet (EDP)) to the GW every 10 seconds. The payload size varies depending on the sensing activity, but in our case it is always lower than the limit of a 100 character payload to fit in a single packet. As an exception to the above sampling rule, environmental nodes check their motion sensors (PIR) every 2 seconds and send a PIR Data Packet (PDP) each time motion is detected. This was required for schools to be able to detect motion in a more rapid manner, instead of waiting several minutes to see an average of multiple motion sensor values. 2 https://github.com/CTI-ru1/sensorApps/tree/master/Arduino/Routing

B. MIGRATING FROM XBee TO A NEW NETWORK TOPOLOGY
During the course of the GAIA project, it became obvious that the use of XBee modules for the development of our IoT infrastructure had become far too expensive compared to the past, mainly due to the limited number of XBee modules available in the market (XBee S1 with a price from 20.3e from a US provider and actually discontinued [59]) to 27.5e from an EU provider [60]); XBee S1 Pro with a price from 33e (from US provider [61]) to 39.20e (from EU provider [62]). As we evaluated other solutions that appeared in literature and the maker communities, we came across the LoRa and LPWAN networks that had gained a lot of popularity. Also, the LoRa modules available on the market had significantly lower cost compared to XBee modules resulting to a lower overall cost for each infrastructure. By using LoRa in a more conventional LPWAN manner (low data rates and infrequent transmissions) could not lead to similar performance as 802.15.4 for our use case, so we designed and implemented the protocol presented in the next section at a lower network level. Finally, we wanted to be able to operate our system without the need to maintain a large private LoRa network, or depend on external communities and platforms like The Things Network 3 for the operation of our network.

C. LoRa NETWORK TOPOLOGY
Our LoRa nodes use a single-hop topology (Figure 3) (tested inside 3-floor concrete-built buildings), thanks to LoRa communication range characteristics. The IoT nodes communicate using the Grove LoRa 868MHz [63] modules (with a price of around 15e) with a LG01-N Single Channel LoRa Dragino gateway [64] (with a cost of around 46e), suitable for small-scale LoRa networks. Both devices utilize the RF95 SX1276 LoRa module [65]. We used this specific module due to cost and size constraints. This choice essentially meant that we could not use LoRaWAN, since the devices' constraints (Arduino) in computational resources do not allow us to do so. Therefore, we implemented a custom solution, so that the GW coordinates the communication with every node to guarantee that nodes do not occupy the medium at the same time. This implementation is necessary to avoid the interference due to the ALOHA MAC protocol used in LoRa and guarantee no interference between the nodes at the same network. The network is created by the GW announcing itself via broadcast messages. The new nodes reply to the broadcast with a connection request which, if accepted by the GW, is acknowledged via a confirmation message. ALOHA with a randomized delay is used by the nodes to answer to the GW network announcement.
Once the network is set up, the GW then sequentially requests data periodically from each node with a Data Request Packet (DRP). The nodes reply with a Data Packet (DP) consisting of the sensor measurements. The request rate of the GW is configurable to adjust both node and GW transmissions to the requirements of ToA EU regulations. In this case, the node samples the PIR sensor between GW requests and includes the motion sensing information in the DP to avoid creating overhead. If a reply is not received or the reply is corrupted, the GW can repeat the DRP up to three times for each node. We implemented our own CRC method at the network level to detect message corruption, instead of using the LoRa module functionality at MAC level. The network is refreshed every 15 minutes; on each refresh, new nodes can be attached to allow the extension of the deployment without reconfiguration of the GW nodes.

V. DATASET DESCRIPTION
In this section, and before proceeding with the description and analysis of the results from our comparative study, we present and discuss the dataset produced from the operation of the installations that we base our results on. Overall, as mentioned above, in GAIA we have implemented a large-scale IoT deployment comprising various types of sensors (power consumption, temperature, light, particulate matter, etc.) across school buildings in 3 European countries (Greece, Italy and Sweden). A typical installation [8] in such a school contains 6 environmental nodes and a couple of power measurement nodes. In this work, we focus on the network level, since we are interested in assessing the performance-related aspects of IEEE 802.15.4 versus LoRabased deployments. We thus omitted the part of the measurements related to actual environmental and energy-related aspects.
The dataset we provide contains network and quality of service measurements from 6 different school buildings. In these buildings we have deployed 8 individual networks consisting of a total of 49 sensor devices. Out of these 8 networks, 5 are based on LoRa and 3 are based on IEEE 802.15.4. In Schools A, B and C we used the existing LoRa based infrastructure while in Schools D, E and F the existing IEEE 802.15.4 network. In order to compare these two networks in the same buildings, we also deployed two more temporary LoRa-based networks in Schools E and F. The dataset is organized in five experiments we conducted and contains periods of data from May 2019 to March 2020. A total of 92 days of experimentation were devoted to each of the experiments that will be presented in the next sections. The total size of the calculated statistics dataset is 40.2 MB (this figure includes only the statistics for the network and not the actual sensed parameters).
In each LoRa installation, we collect a number of parameters to evaluate its performance and the quality of the communication. As link state metrics we collect the Received Signal Strength Indicator (RSSI) and Signal to Noise Ratio (SNR) for both ends of each gateway-node link as measured by the LoRa SX1276 module. In the dataset, the prefixes gateway_ and node_ denote where the signal is coming from. The payload is the size of the data portion of the message transmitted by the node in bytes. These data are collected from each individual message that reaches our servers where it is averaged into 5 minute aggregates.  As network quality indicators, we provide the ones presented in Table 5. These statistics are collected from the gateway in 15 minute intervals. The polling_cycles is the number of times we have polled the network for data. In each cycle all the nodes known to the gateway are requested for data at least once. messages is the number of valid data packets received by the gateway from a node and publications is the number of data packets successfully transmitted from the gateway to the server. To measure erroneous conditions, we collect the crc_errors and the invalid_packets, which are the number of packets failed the CRC validation and the number of packets that were malformed (i.e., the packet did not adhere to the format utilized in our network) respectively. We also independently collect the number of additional requests sent to a node if a reply was not received in retransmissions. Despite crc_errors and invalid_packets may cause additional requests to be sent to the nodes, these cases are not accounted in retransmissions. In the dataset, we have omitted periods when we were not receiving data from the network due to external factors, such as loss of Internet connectivity.
In our round-robin LoRa network implementation, over the physical LoRa layer, the maximum number of data packets in every 15 minute period depends on the total number of IoT devices connected and, on the specific network configurations in order to guarantee the European limitation for the specific frequency ToA.
We are also interested in the connectivity between the GW and every node in our LoRa network. We quantify the quality of each link by calculating the Packet Delivery Ratio (PDR) for every node. The PDR of the link between node A and the GW can be measured as the ratio between the number of DPs received by the GW from node A, and the number of DRPs sent from the GW to node A. The GW makes one DRP and a maximum of 3 DRP retransmissions per node. We can also calculate additional metrics to further our understanding of the network. The CRC Error Ratio is the ratio of DRPs from the GW which resulted in a corrupted DP being received. Re-transmission Ratio is the ratio of DRPs required to be repeated, either because of CRC errors, malformed DRP or due to not receiving a reply.
For each IEEE 802.15.4 network we collect the statistics presented in Table 6. These statistics are calculated every 15 minutes by analyzing the number and content of the packets received by the server from the gateway. env_messages is the number of messages received by the gateway from a node in the last 15 minute period. Every node in the IEEE 802.15.4 network is scheduled to send data to the gateway every 10 seconds, resulting in a maximum of 90 packets. Because the nodes can also send additional messages when movement is detected we separate those as pir_messages. We also encountered some cases where nodes would re-transmit their data packet despite the gateway having received the packet already, possibly due to MAC-level retransmissions and lost ACK messages. We record these additional messages as pir/env_duplicates.
For comparing our IEEE 802.15.4 and LoRa implementations, we use the Network Delivery Ratio (NDR) indicator. The NDR is defined as the ratio between the measured DPs and the potential maximum number of packets that could be delivered by each node in the network in a time period. We calculate this value in time intervals of 15 minutes. In practice, we try to quantify the end-to-end success of packet deliveries from a Node to the GW, disregarding how the transmissions are routed through single or multi hop network topologies and MAC and LAN protocols.
The collected dataset is available on GitHub [66]. The repository contains 3D representations of the installations to depict the placement of the devices and the extracted statistics for each of the performed experiments and locations.
Our dataset [66] contains the data of the five experiments we discuss: Other researchers working in this field, can access our data to validate our calculations and extend our observations. We believe that such data could allow the rest of the community to test their own approaches, evaluate the performance of their own network installations or possibly experiment with the data using it as input to network simulators.

VI. LoRa NETWORK CONFIGURATIONS COMPARISON
In this section, we provide details regarding the configuration of the LoRa networks utilized in our comparison; this will help better understand the theoretical limits of our setup and how it fared in overall terms within this context. As mentioned previously, duty cycle is restricted in Europe to 1% for LoRa. The data sheet for the SX1276 module [65] provides a formula to calculate the number of payload symbols and the preamble length. Using this, we calculated the ToA for each packet in our scenario, in order to then calculate the maximum allowed packet rate under different LoRa configurations. DPs have a maximum payload of 60 bytes, while for DRPs this is set to 4 bytes. A comparison for each packet type between different network configurations is included in Table 7, showing the corresponding PL, ToA for a single packet in milliseconds and the minimum Period between transmissions in seconds.
The GW requests data from each node periodically by sending a data request packet (DRP). Due to this feature, the maximum packet rate (minimum period between DPs) per IoT node is basically limited by the rate in which the gateway can request data by the LoRa nodes; this, in turn, depends on the total number of nodes in the network, i.e., the gateway has to ''poll'' the nodes and the speed in which it does that affects the overall performance of the network. Table 8 provides some concrete examples of how these factors interplay for two schools with different number of nodes (6 and 7) and what are the theoretical minimums for the time between DRPs and the maximums for the data rate of the IoT nodes, and how these change for different network setups.
In terms of the requirements for our application for packet rate per node, a suitable combination is a Spreading Factor of 7 and a signal bandwidth of 500kHz. Because we prioritized a good sensing data rate from the IoT nodes, in the end we decided to use this configuration in the deployments studied here. However, in this section we provide a comparison of other configurations as well. As mentioned in previous sections, using higher spreading factors allows longer range but results in lower data rate. We continue with comparing network behaviour under two extreme configurations: Configuration 2 gives us higher rate (SF 7, BW 500kHz) while Configuration 1 (SF 9, BW 125kHz) results in longer transmission range. In order to study the network behaviour, we collected the following measurements per node: the number of DRPs from the GW, the number of received DPs, and the number of packets received with CRC errors over a period of 15 minutes.
In the case of Configuration 1, the maximum number of packets received is limited by the ToA imposed by communication regulations, i.e., we have to set the GW request   rate accordingly. As such, Configuration 1 is limited to 12 packets per 15 minutes, per node. Configuration 2 is limited respectively to 193.69 packets per 15 minutes, per node (Table 8). We have to bear in mind that the IoT nodes require certain time overhead to poll their sensors to produce new readings, which incurs extra time overhead in the whole process. We also inserted a time delay of 50ms between successive GW requests, due to the additional processing overhead between the microcontroller and the transceiver. This has resulted in a maximum of 174 packets per 15 minutes for Configuration 2 (Table 9), naturally lower than the theoretical value calculated, while still close enough. Tables 9, 10 and Figures 4c,4d,4b,4a showcase our results produced by our deployment in School A using two network configurations during two different time periods. These include the number of delivered DPs per node, PDR, RSSI, re-transmissions and CRC errors. For both configurations, we notice that the average number for delivered DP rate (Table 9) is higher for the nodes near the GW (1,5,6).
Before our actual experiments took place, we expected to see worse results for Configuration 2, mostly due to selecting parameters for a higher packet rate. We see that the median value for the CRC Error Ratio distribution of each node is higher and with similar standard deviation, with the exception of the closest node (Node 1, CRC Error Ratio graph in Figure 4a). The degradation of network quality is evident at the farthest nodes (3 and 4) from the GW regarding the number of Re-transmissions (Figure 4b), which exhibits higher standard deviation and more frequent and distant upper outliers. In addition, the PDR (Figure 4c) of these nodes is worst than in Configuration 1, with higher standard deviation and more frequent and distant lower outliers. On the other hand, the nodes closest to the GW achieve better network behavior regarding Packet Delivery and Re-transmission Ratios. Furthermore, the RSSI distribution (Figure 4d) exhibits greater standard deviation, entailing less stable signal strength.
Eventually, we observed the cost in network quality that we expected materializing only in the nodes located farther from the GW, while in the nearby nodes we saw an increase in the link's efficiency. This, combined with an increase in the per node packet rate, resulted in a significant increase in the number of sensor measurements produced across the whole network. We did not observe large differences in the per node packet rate among nodes in different places, with the biggest VOLUME 8, 2020  difference being in Configuration 2 among one node (node 3) that had the worst value of PDR (0.86), compared to the best value in the network (0.99) and to its prior score (0.96), but still this means that much more packets are delivered despite the lower PDRs.

VII. OBSERVATIONS ON INTERFERENCE IN LoRa INSTALLATIONS IN SCHOOL BUILDINGS
After having described the limits regarding our LoRa configurations, in this section we proceed with some observations we made regarding interference in our networks, how it degraded performance, and the solutions we utilized to deal with it. The Building complex installation shown in Figure 5 consists of three different school buildings. The distance between School A and School B is 100m and between School B and School C is just 52m. The School A is a one floor building where we installed a LoRa network consisting of 1 GW and 6 Nodes. On the other hand, the School B and School C are 3-storey buildings where we installed a LoRa network consisting of 1 GW and 7 and 8 nodes respectively. Our networks are configured to utilize high transmission data rates, sending to the network at the TABLE 11. Transmission patterns of every node in every network in the inter-interference real scenario in a 15-minute period, as an average of all the data collected at the specific dataset, where polling cycles represent the number of the requests issued by the GW, the messages is the number of messages received correctly from each node, the payload is the payload of the received messages and the rate is the transmission rate per node in seconds.

TABLE 12.
Network transmission patterns. Where the Avg. Messages is the average of the total number of messages sent in the network per minutes, the Avg Payload is the payload average, the Avg. Transfer period is the average of the every node transmission period and, Avg Transfer rate is the average of the transmission rate of all the nodes in the network. maximum data rate permitted by the specific LoRa network configuration. On the specific scenario in School A, every node sends close to 60 Bytes every 5.51 seconds, every 6.28 seconds on School B and, every 8.19 seconds on School C, a total of 10.53KB per minute. Additionally, the requests from the GW add an extra 0.554KB to the network load (Tables 11 and 12).
Initially, we used the same LoRa Network configuration on every school and we observed issues appearing on such a scenario where networks are closely located. The nodes from School C connected through the GW placed in School B. This indicated that part of School C building is covered by both LoRaGWs (Figure 5). Because the two GWs coordinate the network access to the middle independently of each other in our implementation, interference takes place between them.

A. PROPOSED SOLUTION: DIFFERENT SPREADING FACTORS IN EACH NETWORK
Due to the specification of the LoRa physical layer, we can setup the network to ''orthogonalize'' transmissions by using different SFs and provide simultaneous collision free communication. Various studies have assumed a perfect orthogonality among SFs, so that multiple networks with different SFs could simultaneously operate in the same channel.
In our experience, we have continued to observe noticeable interference that affected the performance of our network. Furthermore, adjusting SFs influences the maximum rate and range. In the case of interfering networks, the highest rate configuration can be used in one of the networks only, limiting the other ones to lower data rates. Similar results have been observed also in other studies of the LoRa SF orthogonality [67].

B. WORKING SOLUTION: DIFFERENT FREQUENCY CHANNELS FOR EACH NETWORK
Our use case working solution uses a different CF (Carrier Frequency) for each LoRa networks present in the close network scenario. The CF determines the central transmission frequency in operating frequency range of the module. Our use case prioritizes reaching the highest transmission rate per node, thus the Configuration 2 (SF 7 and BW 500 kHz) is used. The CFs used at the close network environment are: 868.6MHz in School B, 868.1MHz in school School A and 868.0MHz School C to avoid overlapping frequencies between the closest buildings. The NDR for every node on each school building under our working solution implementation is show in Table 13. The specific values collected during a month shows the stability of our proposal for close network scenario in our use case. VOLUME 8, 2020

C. INTER-SF EXPERIMENTS
In order to study how the Inter-SF interference affects the LoRa network, we performed a number of experiments on a real-world small-scale installation in an office space. To this purpose, we analyse the link quality between the LoRa GW and a LoRa node in presence/absence of a secondary LoRa network under different configurations. To quantify the quality of the LoRa link we utilize the messages, re-transmissions, FIGURE 6. Reference default LoRa link quality (SF=7, BW=125 MHz, CF=868.0 MHz), where msg is the number of data packets received correctly, ret is the number of additional requests sent because a reply was not received, crc is the number of additional requests sent due to CRC errors in the received msg, and cyc is the number of times that the link has been polled for data.
CRC errors and polling cycles network quality metrics as they are described in the dataset (Section V).
As the basis of these experiments, we used the default module configuration, SF 7, BW 125 kHz and CF 868.0 MHz which link quality is being shown in Figure 6. Then, we expose the basic configuration LoRa Link under diverse interference sources and study their effect on the LoRa link quality indicators (Figure 7). Firstly, we consider the inter-SF interference by set a second LoRa link configure at the same frequency with different SF (SF=9, BW=125 MHz, CF=868.0 MHz) under diverse Bandwidth (Figure 7a and Figure 7b). Secondly we considered a LoRa link that transmit in different frequency but with the same modulation in terms of SF and BW, as the interference source. The (Figure 7c and Figure 7d) shows the statistics for the link working at 868.3MHz with CF = 300kHz and at 868.1MHz with CF = 100kHz. We observe that we achieve a bigger number of message received correctly with lower re-transmission number . LoRa Link quality in presence of interference sources (SF=7, BW=125 MHz, CF=868.0 MHz), where msg is the number of data packets received correctly, ret is the number of additional requests sent because a reply was not received, crc is the number of additional requests sent due to CRC errors in the received msg, and cyc is the number of times that the link has been polled for data.
when the second network source of interference works in different frequency (degradation of 8% 7c to 10% 7d) than when the networks are set to send data with different modulation (different SF) (degradation of 16% 7a to 25.3% 7b). In addition the inter-SF interference effects in the link quality is highly influenced by the modulation BW selected, as higher the BW higher the quality link degradation (Figure 7a and Figure 7b).
The CF separation between both LoRa networks was chosen to guarantee the absence of overlapping in relation with the BW section at least in theory. In both experiments, quality degradation is shown in Figure 7c and Figure 7d, the CFs distance ( CF) is bigger than the half of the BW but either way the biggest distance in terms of CF results in lower quality degradation. Furthermore, we can observe that the network degradation is bigger when the networks that suffer inter-interference are utilizing neighboring frequency bands (Figure 7d), due to the power leakage in the channel filters. However, even in this case, the networks achieve a better transmission rate with a lower number of re-transmissions than in the multiple SFs scenario.
Different SFs are used in LoRaWAN networks automatically to adapt the better rate transmission for different ranges without interference. On the other hand, based in our results, the use of different LoRa frequency channels allows us to avoid interference more efficiently in closely located deployments thus avoiding collisions under the ALOHA physical LoRa layer.

VIII. COMPARING NETWORK BEHAVIOR IN SCHOOL BUILDINGS
In this section, we proceed with presenting the core of our comparative study of the two technologies. We first discuss network saturation in our IEEE 802.15.4 deployment, followed by an analysis of experimental results produced first by having the 2 types of network inside different school buildings, and then in the same school building for an even more direct comparison. In each of the following evaluation VOLUME 8, 2020 scenarios, the placement of the devices inside the buildings was done after a thorough survey of the area, to identify the most suitable placement locations to achieve the best communication and highest reliability of the deployed infrastructure. Environmental Data Packets (EDPs) are transmitted every 10 seconds while PIR Data Packet (PDPs) when motion is detected in the case of the IEEE 802.14.4 networks only (in the LoRa networks no additional PDPs are transmitted). On the other hand, the LoRa GW requests data from each node at the higher data rate allowed by the European regulation duty cycle for our specific network configuration (for Config. 2 with transmissions every 5.52 seconds on 6-node networks and up to 9 seconds on 8-node networks, due to the ToA limitations on the LoRa GW as we explained in Section VI). To quantify the effects of the independent PDPs, we use their ratio against the observed maximum of the aggregation. Figure 8 shows clearly how during school hours the PDPs can cause a notable decrease in the number of EDPs, due to the saturation of the network at peak of PDP Ratio. The number of PDPs exhibits a maximum, because of the peak of activities in the school building, the number of EDPs decreases below their average. In addition, the maximum number of EDPs is observed when the number of PDPs is zero (Table 14), effectively when there is no movement detected. Our decision to include real-time motion detection to the network, can potentially be a hindrance to the stability of the EDP rate of our network.

B. DISCUSSION -COMPARISON INSIDE DIFFERENT SCHOOL BUILDINGS
In this section, we compare the two technologies using a LoRa and an IEEE 802.15.4 deployment in 2 different school buildings. School A hosts a deployment with a total of 6 LoRa nodes; nodes 3 and 4 are located at the farthest positions, while nodes 1 and 5 are the ones located nearest the GW. School D hosts an IEEE 802.15.4 network with 6 nodes; node 6 is the farthest from the gateway and node 5 the nearest. To compare them we quantify the ''quality'' of these networks using the Network Delivery Ratio (NDR) from our dataset (Section V).
In the LoRa network using Configuration 2 with 6 nodes, the network can achieve the delivery of a maximum of 174 packets per node. On the other hand, our IEEE 802.15.4 network can theoretically provide a maximum of  90 EDPs and an number of PDPs per node. The dataset consists of NDR measurements collected during a period of both weekdays and weekends for School A (LoRa) and School D (IEEE 802.15.4) can be seen in Table 15 and Figure 9.
We see that the best network quality in terms of NDR is in School A (LoRa) under Config. 1, with the maximum number of Data Packets per node observed in a 15-minute period being 12. The network in School D (IEEE 802.15.4) exhibits better NDR than School A (LoRa) under Config. 2 with respect to the averages for every node, but with the exception of the farthest one that achieves a delivery of 20% of the generated packets. On the other hand, the network in School A (LoRa) under Config. 2 achieves a more stable NDR across the whole installation, including the nodes that are located far from the gateway, with successful packet deliveries between 86% and 89% for every node and a significantly higher delivery rate. In order to achieve a packet rate that is equivalent to School D (IEEE 802.15.4), we must use Config. 2 in School A (LoRa). Although in this case the network quality of Config. 2 fluctuates, as is evident by its significant standard deviation, it achieves better NDR in distant devices (Table 15 node 6).
From these numbers, we can make the conclusion that in the multihop network topology in IEEE 802.15.4, established in order to cover the same area as LoRa, actually decreases the packet rate of the nodes placed at the edges of the network. This is mainly due to poor connectivity in the IEEE 802.15.4 frequency band even for devices placed in nearby rooms. The buildings in which we installed the devices have thick concrete walls that some times affect even full power WiFi  signals. As a result, although we have added devices for nodes to communicate via multihop routes, there are still cases where some steps of the path suffer from low packet rates that cannot be avoided. In comparison, LoRa star topology offers same or better coverage, with a more stable data rate on all nodes ( Figure 9).

C. DISCUSSION -COMPARISON IN THE SAME SCHOOL BUILDING
After comparing the two technologies using deployments in different buildings, we continued our experiments by collecting data from installations within the same building, i.e., we installed both LoRa and IEEE 802.15.4 networks at the same school building at exactly the same spots, working under the same environmental and user conditions for 20 days. The IEEE 802.15.4 modules works at 2.4 GHz and the LoRa module work at 868 MHz in order to avoid possible interference's between both communication technologies [46]. For this specific experiment, we have improved the LoRa implementation to achieve a bigger data rate. To guarantee the ToA restriction, the LoRa configuration Conf. 2 (SF 7, BW=500 kHz) is used, achieving a maximum number of 232 messages per node in a 5 nodes installations, and a maximum of 193 messages per node in a 6 nodes installation for a 15 minute period.
The first building under study was a single floor building with 900m 2 interior surface and 3m interior high and, one external metal prefabricated building of 40m 2 and 2.5m high. All 5 sensor devices are installed in the same level. Each device communicated with a single gateway node in both configurations. The installation is presented in Figure 10. The sensor devices are depicted as red boxes and the gateway node as a cyan box. The 5 locations where we installed sensor devices were 3 classrooms (yellow, red and blue rooms) the main hall of the building (cyan room) and an external classroom on the one side of the building (green room) in a prefabricated building. 236 students are using the building at the mornings and a more reduce number of students using the building from 14:00 until 16:00.
The second one was a two story building with a 1300m 2 interior surface and 4m interior high each one with one central wide staircase. A total of 6 sensor devices are placed in both levels (2 on the ground floor and 4 on the first floor). The single gateway node was installed on the first floor of the building in both cases. The installation is presented in Figure 11. The 6 locations where we installed sensor devices were 5 classrooms (yellow, red, blue, green and cyan rooms) and a utility room (purple room). The number of students of the school using the building at morning ours is 209 at the moment of the experiment, using mostly the 1 floor. Also, the school is used by a varying number of students every afternoon, principally in the ground floor.
We aim to compare the quality network of both implementation under the same using conditions in the same real buildings. The NDR of each link for both IEEE 802. 15.4 and LoRa are represented in Figure 12 and Figure 13 with the same color as the classroom where the specific node is placed ordered according to increasing distance to the GW. The IEEE 802.15.4 Networks achieve an average delivery ratio between 98% and 91% in School E and 98% to 90% in School F. We can observe that on both schools the farthest nodes exhibit the worst NDR in the IEEE 802.15.4 networks while the LoRa networks shows a more stable link quality (95%) across the buildings installation independent of the location. On the other hand, the NDR statistics calculated for the LoRa networks over the whole period of measurements are a little lower than the corresponding IEEE 802.15.4 numbers for every link in the closest positions to the GW (Figure 12a and Figure 13a).
During the analysis of the LoRa networks, we witnessed fluctuations in the polling cycles of the networks, which upon further investigation was not related to the networks themselves. This periodical behaviour manifested as the LoRaGW requesting a lower number of DPRs during the night that affected the collected statistics. Further investigation indicated that this behaviour was related to the time required from the gateway to establish a connection to the server and publish the data collected from each node rather than with the LoRa networks themselves. In order to dismiss this effect from the statistics regarding the LoRa link quality, we also calculated the NDRs based on a maximum polling cycle number inside a period of 8 hours around each data-point Figure 12b and Figure 13b. Under this specific study the NDR for every LoRa node shows a small increase in the quality of the link, achieving similar ratios to the best-case quality link of the IEEE 802.15.4 network.

IX. DISCUSSION -COMPARISON OF COSTS
After having compared the 2 technologies in terms of performance, we now compare our LoRa and IEEE 802.15.4 implementations in terms of cost for our specific use case, in order to highlight the advantage of LoRa in this area as well. LoRa-based IoT devices are currently cheaper than ones based on IEEE 802.15.4, mainly due to the lower cost of the LoRa module (15-20e) in comparison with XBee S1 pro module (35-45e). There is also a small additional cost for the XBee custom PCBs we used compared to the LoRa-based devices, as well as an additional power regulator required. We use the exact same bill of materials for the two devices, save for the parts related to the use of either LoRa or IEEE 802.15.4. The total indicative cost for our devices installed in our network is shown in Table 16. In addition to the LoRa devices being cheaper than the IEEE 802.15.4-based ones, the LoRa-based implementation is able to cover the entirety of the desired sensing locations inside a school building without extra repeater devices, which is often not the case with IEEE 802.15.4, as discussed in previous sections, where we had to add more sensor devices to also relay messages from others.
We now give an example for a 3-floor school building (School B), with the target locations spreads over 3 floors, indicated with a red box, in the school map in Figure 14, as well as a gateway located in the ground floor, indicated with a blue box. A LoRa star network topology, with the configuration that provide the bigger rate, can cover all the target places with just 1 central gateway and one power meter device per location. When using IEEE 802.15.4, it is necessary to use 4 repeaters and extend the tree topology to cover the specific target places, which obviously requires an extra cost. The specific locations for each repeater is indicated with green boxes in Figure 14. The total cost of the installation at this school is 639e when using a LoRa implementation and 1187e when using an IEEE 802.15.4 implementation. We are thus able to reduce the total cost of the installation by 46.17% via using LoRa for this specific building, while still being compatible with our application requirements.

X. CONCLUSION AND FUTURE WORK
Through our work in the GAIA and E2Data H2020 research projects, over the years we have built a large IoT infrastructure scaling across 3 European countries. Since our work was originally aimed at energy efficiency in school buildings, this infrastructure is inside buildings of this type, i.e., public buildings with lots of users, which are also present everywhere and with more or less similar characteristics over Europe. This activity has been implemented over the course of several years, and practical implications have led us to experiment with different wireless networking technologies like IEEE 802.15.4 in the beginning, and more recently, LoRa. As a result, we were able to make a comparative study, both in an empirical/qualitative manner, as well as in a quantitative one, between these two technologies. Although at first glance they do not seem to be competing with each other, there is a growing trend towards using LoRa in cases where technologies like IEEE 802.15.4 were used up till now, like small-scale smart city and IoT applications. Our experience definitely verifies that this is a viable approach that has several practical merits. Although we did not utilize LoRaWAN, and chose to focus on implementing a more custom approach based on LoRa, we believe our study presents interesting insights to the community.
Overall, our results presented in this work show that in the use-case scenario and environmental settings of school buildings, LoRa-based wireless communication can have an advantage against competing technologies, in terms of reliability and complexity of networking. LoRa, with its significantly better communication range, can handle scenarios within multi-floor deployments better, avoiding situations where repeaters and multi-hop topologies are required for other technologies to implement a well-performing network. In scenarios with relatively low bandwidth requirements, it also performs well enough to be competitive with IEEE 802.15.4. In fact, by selecting carefully the network parameters for our deployment, we saw similar or better numbers in terms of performance and reliability. Within our application context, we noticed considerable interference between neighboring LoRa deployments, which can be dealt with efficiently by careful assignment of frequencies to such networks. We were also able to implement a well-behaving and performing solution based on LoRa with lower overall cost compared to other technologies.
With respect to our future work, we will continue our efforts on utilizing LoRa and expand our IoT infrastructure, as well as explore additional practical aspects, such as investigate the use of guaranteed time slots (GTS) in IEEE 802.15.4 for solving the issue of having many collisions. We also plan to continue our work in assembling related datasets and releasing them to the community, since currently there is a relative lack of datasets produced by real-world deployments.