Coverage and Energy-Efficiency Experimental Test Performance for a Comparative Evaluation of Unlicensed LPWAN: LoRaWAN and SigFox

Low Power Wide Area Networks (LPWAN) have emerged as an attractive Internet of Things (IoT) communication option. When deploying a communication network to support IoT applications, large coverage and low power consumption are critical requirements. Despite the fact that existing LPWAN technology solutions promote IoT requirements such as long communication range, energy efficiency, scalability, and low cost, network performance is a major concern. With so many LPWAN technologies available, there is a growing interest in evaluating them. Recent works have presented various comparison studies of LPWAN technologies, but the majority of them have approached the analysis from the standpoint of comparing technical specifications rather than presenting measurement results obtained from network deployment scenarios. We argue that by proposing a comparative evaluation from an experimental standpoint, the comparison discussion is deepened. This paper proposes an experimental comparative evaluation of LoRaWAN and SigFox, two emerging LPWAN technologies operating in sub-GHz Industrial, Scientific, and Medical (ISM) frequency bands, based on coverage and energy-efficiency test performance. The experimental evaluation was proposed first by identifying coverage and energy-efficiency as the two most important design goals for LPWAN applications. Second, by proposing test performance to evaluate those goals, where extensive measurements were made in network deployments, and finally, by highlighting the main performance findings in both networks for comparison purposes. The results show that in a fair-weather test, LoRaWAN outperformed SigFox in terms of coverage, achieving a higher packet delivery rate (PDR $\gtrsim ~80$ %), and having higher radio strength signal (RSSI $\gtrsim $ -110 dBm). Sigfox, on the other hand, shows better energy efficiency with 20% more sent messages under the same test conditions.


I. INTRODUCTION
Communication networks for the Internet of Things (IoT) 22 have gained prominence since the introduction of the IoT 23 The associate editor coordinating the review of this manuscript and approving it for publication was Ghufran Ahmed .
paradigm. The International Telecommunications Union 24 (ITU) formally proposed IoT in 2012 [1], [2]. IoT appears 25 to be the most recent evolution of the Internet, which has 26 progressed through five major milestones: First, one-on- 27 one communication, also known as pre-Internet. Commu-28 nications or content-Internet via the ''www''. Second [26], [27], [28]. The numerous emerging solutions for long-90 range in IoT wireless networks with low power consumption 91 have attracted the interest of standardization bodies such as 92 the 3GPP and the IEEE Standard Committee. NB-IoT and 93 the IEEE 802.11 standard, for example, developed common 94 technologies that can be used in a variety of scenarios [29]. 95 The works found in the literature have presented several 96 comparative analysis of LPWAN technologies, but to the 97 best of the knowledge of the authors, most of them have 98 limited the evaluation and comparison only by defining a 99 common framework with a systematic approach based on 100 network features and requirements, without presenting mea-101 surement results obtained from network deployment sce-102 narios [18], [30], [31], [32], [33]. It is always important 103 to define a common framework when comparing technolo-104 gies; however, additional questions arise in terms of how 105 to measure performance and test the technologies in terms 106 of the defined requirements. Only few papers have pro-107 posed an experimental evaluation of LPWAN technologies 108 [34], [35], [36], [37], however, they have been oriented 109 mostly to the exploration of a single technology and focused 110 on one particular performance analysis. As for instance, the 111 authors in [36] and [37] propose experimental evaluation 112 set-ups of LoRaWAN networks for coverage performance 113 in urban or rural scenarios. The work in [34], analyzes the 114 scalabillity of a realistic SigFox communication model by 115 generating SigFox traffic using Software Defined Radios. 116 The work in [35] does include extensive measurement results 117 of NB-IoT, SigFox, and LoRaWAN network deployments 118 in the cities of Brno and Ostrava with plenty of Base Sta-119 tions, however measurements results are oriented to propose 120 enhancement of selected LPWAN radio propagation models 121 in urban environment, without a comparison perspective. 122 This paper proposes an experimental comparative evalua-123 tion of LoRaWAN and SigFox in urban and rural scenarios 124 based on coverage and energy-efficiency test performance, 125 as illustrated in Fig. 1. LoRaWAN      In addition, the authors in [37] presented an experimental 249 evaluation of LoRaWAN for a wildlife monitoring applica-250 tion in a forest vegetation area. The PDR, RSSI, and Signal-251 to-Noise ratio (SNR) were tested as experimental network 252 metrics for performance evaluation with different payload 253 length, SF, and CR. The experimental evaluation is limited to 254 LoRaWAN and is conducted in a forest environment, where 255 they achieved a maximum communication range of 860 m 256 with an SF of 12.

257
The authors of [34] presented a scalability analysis 258 of the SigFox communication protocol under large-scale 259 high-density conditions using a SigFox traffic generator 260 implemented via Software Defined Radios (SDRs). When 261 360 orthogonal channels are available, the structural scal-262 ability obtained in the proposed scenario is approximately 263 100 sensor nodes. The experimental evaluation is limited 264 to SigFox, and it is a lab experiment without network 265 deployment.

266
Finally, the authors of [35] presented a case study for 267 selecting accurate radio propagation models for Narrowband 268 IoT (NB-IoT), LoRaWAN, and SigFox LPWAN technolo-269 gies. Based on experimental measurements, they propose an 270 improvement to selected propagation models. Despite the 271 deployment of experimental network setups for extensive 272 measurements, the goal of the paper is to cross-validate radio 273 propagation models in two cities. In other words, their main 274 contributions are aimed at providing a methodology for fine-275 tuning propagation models for LPWAN technologies based 276 on experimental results, which falls outside the scope of an 277 experimental comparative evaluation.

278
To summarize, none of the works found in the literature 279 propose an experimental comparison evaluation of different 280 LPWAN technologies based on network deployments. The 281 related works in terms of compared LPWAN technologies, 282 comparison perspective, performance analysis, and evalua-283 tion scenarios are summarized in Table 1.

285
Many works have presented the goals, requirements, and 286 features of LPWAN design [31], [32], [33], [42], [43]. They 287 all agreed that LPWANs are the best network solution for 288 large-scale IoT system deployments over large areas due to 289 their energy-efficient working schemes, low-cost and low-290 complexity end-devices, low data rates, and high latency. 291 Even though design considerations and requirements differ 292 in some ways, they can be classified as general design goals 293 or considerations. The authors of [43] defined application 294 requirements based on LPWAN coverage, capacity, cost, low 295 power operation, and enhanced characteristics. Coverage has 296 been identified as being fundamental to almost all of the 297 identified main applications, followed by low power opera-298 tions primarily driven by the lack of electric power supply 299 in remote locations, such as smart agriculture and farming, 300      These two requirements are discussed further below.

320
LPWAN are intended to work over long distances as 321 wide-area networks, which means that their communication 322 schemes must allow end-devices to efficiently deliver 323 messages over a few kilometers in urban areas and tens of 324 kilometers in rural areas. When compared to mobile cellular 325 networks, the communication target of LPWAN is increased 326 by 10-40 km in rural zones and 1-5 km in urban zones, 327 with a link budget increase of +20 dB. Some applications 328 may require connectivity in indoor environments, particularly 329 underground and basement locations, which are generally 330 difficult to access. In comparison to higher ISM frequency 331 bands, LPWAN achieves long-range communications with 332 robust and reliable characteristics by using Sub-GHz fre-333 quency bands. In any case, coverage must be evaluated not 334 only from the perspective of the link budget, but also from the 335 standpoint of the package delivery rate (PDR). We considered 336 a target coverage of 5 km in urban areas and 10 km in rural 337 areas, with a PDR of more than 90%.

339
Many IoT applications require end-devices to be ubiquitous; 340 in particular, some applications require devices to be remotely 341 located, so they must be battery powered and not recharge-342 able. As a result, low power consumption must be ensured 343 through the use of low data rate modulation techniques. In this 344 regard, the authors of [42] set a target battery lifetime of 345 10 years for end-devices. However, battery life is highly 346 dependent on message sending rate, which is directly related 347 to the type of IoT service, whether critical or massive.  LoRAWAN is a proprietary LPWAN technology based on 362 the LoRa physical (PHY) layer that provides wide cover-363 age whereas consuming low energy and transmitting low 364 data rates. Semtech, IBM, Actility, and Microchip created 365 LoRaWAN in North America. The LoRaWAN network is a 366 single-hop network in which end-devices or motes connect 367 directly to a LoRAWAN base station that acts as a gateway to 368 the information server.

369
The LoRa PHY uses Chip Spread Spectrum (CSS) modula-370 tion. CSS is a Direct-sequence spread spectrum (DSSS) sub-371 category that uses controlled frequency diversity to recover 372 data from weak signals, even near the noise level. CSS modu-373 lation was widely used in military communications due to its 374 low transmission power requirements, resistance to channel 375 degradation, multi-path, fading, Doppler effect, and jamming  The SF is defined in (1), which is related with spread 389 bandwidth B, the symbol rate R s and the chirp duration . This is a significant 431 limitation for both LoRaWAN and networks using unlicensed 432 frequencies. Therefore, the frequency channel selection must 433 adhere to the maximum duty-cycle and implement pseudo-434 random channel-hopping at each transmission [53].

435
By utilizing a pseudo-random frequency hopping 436 approach, an end-device is able to transmit at any time in 437 any open sub-channel. When this occurs, the end devices 438 operate at their highest bandwidth within the constraints of 439 the duty cycle limitation. When the transmission power is 440 greater than 20 dBm, the gateway administrator supports up to 441 10 4 end-devices. Forward Error Correction (FEC) is a method 442 used by LoRaWAN to repair errors. The trade-off between 443 coverage and message duration (i.e. Time over the Air -ToA) 444 determines the data rate. High data rates not only increase the 445 ToA but also carry additional data for interference protection. 446 In terms of energy consumption, authors in [27] has 447 addressed the energy consumption of LoRaWAN end-devices 448 for the evaluation of multi-hop bidirectional communication 449 in a wide-area application. Results of their experimental set-450 up demonstrates a network coverage of 150 m with only 451 6 end-devices, achieving a potential node life-time of 2 years 452 with batteries of 5400 mAh capacity, transmitting every 5s 453 and reaching a reliability above the 80%. where R is the distance in meters between the transmitter and 478 receiver, λ is the free-space wavelength in meters and CG dB 479 is spreading coding gain, given by 2.  Similar to a LoRaWAN deployment, when we have a data 538 link with a line of sight (LoS), the signal-to-noise relation 539 (SNR) is computed in this case, with the bandwidth occu-540 pation being much less due to the use of UNB modulation, 541 which is 100 Hz. The noise floor is therefore set to N dBm = 542 −154 dBm +NF dB . The signal-to-noise relation is given by 543 (4) using the same considerations for free space as in the 544 LoRaWAN network.
The received power must be P Rx ≥ −140 dBm with an 547 end-device transmission power P Tx ≥ 14 dBm allowed by 548 the ETS regulation [57], when taking the same SNR threshold 549 as before of 8 dB and a link margin of 4 dB. This results in a 550 coverage area of thousands of kilometers. In reality, authors 551 in [11] reports a range of 63 km.

554
With the goal of having comparison between LoRaWAN and 555 SigFox, Table 2 summarizes the main specifications of both 556 technologies.

TEST PERFORMANCE
559 LoRaWAN and SigFox were tested in both urban and rural 560 scenarios in this study. The Received Signal Strength Indica-561 tor (RSSI) was used to test the communication range, as well 562 as the Packet Delivery Rate (PDR) and energy efficiency of 563 its end devices when transmitting in similar environments. 564 The LoRaWAN network was deployed using a Multitech 565 gateway, which can also function as a network server due to a 566 pre-installed application called node-red, which allows direct 567 interaction with received messages via javaScript. As end-568 devices, Pycom's LoPy/LoPy4 and SiPy modules were used 569 and programmed in Python, with received messages stored in 570 a SQL database. The configuration parameters of the network 571 deployments are presented in Table 3.

573
The tests for the urban scenario were conducted around 574 our University campus, where the average building height 575 is around 50 meters. A LoRaWAN and SigFox base sta-576 tion were installed on the terrace of our faculty build-577 ing, approximately 30 meters above the floor level, in the 578 test scenario. The measurements were taken at distances 579 of 100, 200, 400, and 500 meters from the base stations 580 in four quadrants (A, B, C, D) that covered all directions. 581 Larger measurement distances were not considered in order 582 to ensure comparable wireless channel conditions, primarily 583 because the SigFox base station was part of a SigFox network 584 deployment in the city, and thus other base stations located 585 nearby could provide connectivity. Furthermore, more than 586 four static end devices were tested sending uplink messages 587 in each radio. Using both technologies, over 14000 uplink 588 VOLUME 10, 2022   network is shown in Fig. 2, where the low building density 609 around the base station is clearly visible. It is also worth not-610 ing that quadrants B and C are primarily agricultural harvest 611 areas.

613
Different tests were designed for the performance evaluation 614 of both networks in urban and rural scenarios, based on the 615 main features, requirements, and design goals of the network 616 considered in III.

618
The Received Signal Strength Indicator (RSSI) measurement 619 of each uplink message from the various end-devices situated 620 around the base stations for both urban and rural scenar-621 ios served as the foundation for the communication range 622 test. Based on the RSSI of the received messages, the test's 623 objective was to assess the radio coverage provided by the 624 network's base station.

625
In order to calculate the packet delivery rate (PDR), the 626 received message rates in both base stations were compared 627 to the total amount of uplink messages sent from the end 628 devices. This allowed to compare the two network technolo-629 gies in the proposed urban scenario and demonstrated how a 630 LoRaWAN network behaved in different radio environments 631 in terms of packet losses.

632
In the urban scenario, 11 static measurement points at 633 various angles in the four quadrants were used for the first 634 radio of 100 m, followed by four points at 200 m, 400 m, 635 and 500 m. In the case of LoRaWAN, the Base Station 636 (BS) intended to receive 250 messages from each end-637 device located at each point, whereas the SigFox BS was 80 638 power was set to 0 dBm, and end-devices in the LoRaWAN 674 network are activated via Over-The-Air Activation (OTAA), 675 with the frequency plan AU915. Furthermore, the SF was set 676 to 7 by default, with an uplink bandwidth of 500 kHz with an 677 operating frequency in the band of 915 MHz.

679
End-devices in LPWAN technologies like LoRaWAN and 680 SigFox are typically in sleep mode whenever an application 681 requires them to be, which minimizes the amount of energy 682 consumed. The end-devices were configured to send the same 683 message for this test. In this case, it was guaranteed that the 684 message would be sent, regardless of whether the base station 685 successfully received it. Every 30 seconds, messages are sent 686 and registered until the battery in the device is completely 687 depleted. Once the uplink message has been sent, each end 688 device's voltage battery level will be measured throughout the 689 process. For this test, an automatic data-logger based on an 690 Arduino platform with an SD shield was integrated, where 691 all time and measured data were saved. The implemented 692 software of the data-logger stores the time stamp obtained 693 from the microcontroller's Real Time Clock (RTC) and the 694 ADC reading regarding the measured battery voltage.

695
Two 3.7 V Li-Polymer batteries with capacities of 696 4400 mAh and 1800 mAh were used for the test. 697 In unidirectional transmission mode (Class A for LoRaWAN), 698 with a payload of 12 bytes for each uplink message sent 699 every 30 seconds, LoRaWAN and SigFox end-devices were 700 configured. The re-transmission message rate of 3 was also 701 considered in the case of SigFox. For statistical validation, 702 the test was repeated several times.

705
The urban and rural scenarios described in section V were 706 implemented. The outcomes and analysis of the proposed 707 performance test are presented in the paragraphs that follow. 708   conditions, the results show that LoRAWAN end-devices 747 outperform SigFox end-devices in terms of interference.

748
The measured RSSI values of the received messages in 749 the LoRaWAN BS in the rural scenario are shown in Fig. 5, 750 where measurements were carried out until 11 km from 751 the BS. In this context, the results were compared to RSSI 752 measurements obtained with a similar LoRAWAN network 753 deployment in an urban scenario with a distance from the BS 754 of up to 3 km. In general, the results show that RSSI values 755 are higher in the rural scenario than in the urban scenario, 756 as expected. When calculating the link budget in the urban 757 scenario, shadowing and fading effects are clearly visible 758 when compared to the rural scenario, where wireless channels 759 are much more dispersed. In fact, the communication range 760 reach in the urban scenario was 3 km with a PDR of less than 761 1% and an RSSI of around -120 dBm based on PDR results 762 in both scenarios shown in Fig. 6. 763 Other conclusions drawn from the results in Fig. 6 are 764 related to the distance at which PDR decreases significantly 765 for both scenarios; thus, in the urban scenario, PDR drops 766 by 60% above 1 km, whereas in the rural scenario, this 767 occurs above 4 km. After this distance, the reliability is nearly 768  SigFox end-device, in fact, this represents that, before the 807 complete discharge of the battery, the total number of sent 808 messages in the case of SigFox reached up to 1276 messages 809 while LoRa reached 1060 messages. The results are similar 810 in the test with a higher capacity battery (4200 mAh), which 811 is also shown in Fig. 7, where the discharge profiles are 812 maintained for both devices; the only difference with respect 813 to the previous case is the total number of sent messages 814 reached for each end-device, which was 2743 in the case 815 of SigFox and 2237 in the case of LoRaWAN. Based on 816 the previous findings, we can conclude that SigFox technol-817 ogy is more energy-efficient than LoRaWAN technology in 818 general.

819
In general, the end devices in SigFox and LoRaWAN are 820 in sleep or standby mode for the majority of the time, except 821 when the application requires it, which reduces the amount 822 of energy consumed. A LoRaWAN end device, on the other 823 hand, consumes more power due to synchronous communi-824 cation, as it invests in the transmission of some additional 825 messages in order to connect with a BS.

827
In this paper, we propose, for the first time, an experimen-828 tal evaluation between LoRaWAN and SigFox, two repre-829 sentative LPWAN technologies that operate in unlicensed 830 frequency bands. This was accomplished by first selecting 831 coverage and energy consumption as the two most important 832 design requirements in the network deployment for LPWAN 833 applications based on criteria found in the literature by var-834 ious authors. Then, performance test were proposed to eval-835 uate coverage and energy efficiency which can be adapted 836 for different LPWAN. Finally, in order to apply oriented 837 performance tests with extensive measurements in different 838 outdoor locations covering line and non-line of sight affected 839 by different obstruction and multipath propagation environ-840 ments, urban and rural scenarios were proposed for obtaining 841 performance metrics for the analysis.

842
According to the findings of this comparative study, the 843 achievable performance of LoRaWAN network technology 844 can greatly vary depending on the deployment scenario, 845 which can be reduced from more than 10 km to less than 846 3 kilometers with a reduction of Packet Delivery Rate (PDR) 847 from more than 90% to less than 40%. Despite the fact that 848 our results are consistent with the communication ranges 849 stated in the specifications, it is evident that measured ranges 850 are significantly shortened compared to the reported standard 851 communication ranges for both technologies in an environ-852 ment with obstructions over a distance of several kilometers. 853 In accordance with the measured RSSI of the signal, which 854 in the case of LoRaWAN was higher than SigFox at least in 855 5 dB for all distances, our results also show that LoRaWAN 856 outperforms SigFox in an urban environment in terms of cov-857 erage, obtaining higher PDR. The SigFox results, in contrast, 858 clearly demonstrate a better energy efficiency operation that 859 consistently reaches at least 20% more of sent messages. As a 860 third design goal with a significant effect on the network's 861 VOLUME 10, 2022 performance, future work might examine the scalability of engineering from Pontificia Universidad Javeri-