X-LoRa: An Open Source LPWA Network

Low power wide area (LPWA) networks, which enable the energy-constraint devices distributed over wide areas to establish affordable connectivity with the Internet, have attracted a lot of attentions in recent years. Among them, Long Range (LoRa) network becomes one of the most promising system and motivates a plethora of applications. In order to deal with the challenges of managing various applications and supporting massive LoRa devices, this paper proposes a fleXible LoRa system, i.e., X-LoRa, with the features of easy deployment and open source. X-LoRa divides the operations of LoRa network server into several decoupled functional modules according to the microservice design principles. Meanwhile, the real-time streaming data pipelines are utilized as message queue for interaction between modules to guarantee the system performance and scalability. The architecture and implementation of X-LoRa is discussed along with the technical features as well as the functionality of each module. Finally, a few experiments are conducted to demonstrate the good performances of X-LoRa.


I. INTRODUCTION
With the development of low power wide area (LPWA) technologies, the emergence of various Internet of Things (IoT) applications is changing the living way in the world. In particular, it is estimated that LPWA units will grow from 109 million in 2017 to 339 million in 2025, with total rising from 924 million Euros in 2017 to 2,251 million Euros in 2025 [1]. Due to its unique technical advantages and features, Long Range (LoRa) system has attracted significant attention from industry and academic in recent years. LoRa operates in the unlicensed Industrial Scientific and Medical (ISM) band and offers the long-distance connectivity to the low-power devices. LoRaWAN, which is specified by LoRa Alliance, defines the communication protocol and system architecture for LoRa networks [3]. There are many noteworthy properties of LoRa networks, such as long communication distances, low energy consumption and low costs, etc. In addition, LoRa nodes usually have very limited computing capabilities and operate on battery power. Therefore, the network servers of LoRa networks are responsible to have sufficient ability to provide the essential services for massive LoRa nodes.
There already have a number of works on LoRa networks. Firstly, some of them focus on comparing LoRa with other LPWA technologies such as Sigfox and NB-IoT [2], [4], [5]. Then, the performance of LoRa networks in specific scenarios are analyzed as well as the capability [6]- [8]. Moreover, a plenty of attention has been paid on the implementation of the specific applications by LoRa technology [9], [10]. Also, several studies explore some algorithm issues in LoRa networks using a simulated approach such as Adaptive Data Rate (ADR) scheme [11], [12]. Besides, the realization of LoRa platform based on LoRaWAN specification is very essential for field deployment. However, most of these platforms are inflexible and difficult to customize new features to meet the needs of diverse applications and scenarios in IoT.
In this paper, a new architecture for LoRa system, called as fleXible LoRa (X-LoRa), is proposed to facilitate the implementation and management of private LoRa networks. The completed X-LoRa primarily consists of several components, i.e., LoRa node, LoRa gateway, LoRa network server and LoRa application server. The main works of this paper are to propose a feasible solution for LoRa network server as well as LoRa application server. Firstly, we divide LoRa network server into four modules, which are responsible for different tasks and services such as protocol processing and node activation. The design follows the principles of microservice allowing for maximum control and flexibility. Then, LoRa application server is proposed in X-LoRa which can customize the payload structure and encryption for various applications. Moreover, the implementation details of X-LoRa are described as well as the interactive mode, database design and deployment. Aim at achieving low latency, loosely-coupled and good load-balancing, the messaging system based on streaming data is used for data exchange and service invocation between modules. Finally, the proposed X-LoRa has been deployed in urban environments and a series of experiments are carried out to evaluate the performances of X-LoRa.
The rest of the paper is organized as follows. Section II presents an overview on LoRa network model. Section III describes the architecture of X-LoRa. Section IV gives the details on the implementation of X-LoRa. Section V demonstrates and analyses the experimental results. Finally, Section VI draws the conclusions.

II. OVERVIEW ON NETWORK MODEL IN LORAWAN
The network model in LoRaWAN is shown in Fig.1, whose functionality of each component is defined in generalities [14]. In this model, the LoRa gateway forwards the received packets from LoRa end-devices to LoRa network server through an IP back-bone. The LoRa network server is required to process a large number of packets transmitted by multiple LoRa gateways. A plethora of functions are expected to implement in LoRA network server, including checking the legality of end-device and gateway, filtering duplicate uplink packets, preforming security checks, scheduling acknowledgments, performing network management mechanisms and forwarding application data to the target application server, etc. Join server only performs device activation process. In addition, several interfaces are used for]] service invocation between different components. Based on this reference model, the proposed X-LoRa takes advantage of various techniques to achieve high performance and flexibility.

III. ARCHITECTURE OF X-LORA
In order to support the massive number of LoRa devices and facilitate deployment, we propose the architecture of X-LoRa for building private LoRa networks. As illustrated in Fig.2, LoRa node and gateway are fundamental components of the entire LoRa networks since they can perform actions for different applications as well as sending information to LoRa network server. LoRa network server fills the gap between LoRa gateway receiving packets from the nodes to before LoRa application server accepting the information. The network server in X-LoRa is composed of four modules, i.e., Connector, Central Server, Join Server and Network Controller. The functions of each module are introduced as well as LoRa application server in this section. Furthermore, so as to facilitate other developers to enhance more features and promote the further improvement, X-LoRa is open-sourced on the GitHub platform [15].

A. LoRa Node and Gateway
LoRa networks are laid out in a star topology including LoRa node and LoRa gateway. The LoRa node is a sensor or an actuator which can collect information from the environment and connect to the gateway by LoRa transmission. The LoRa gateway forwards all the received radio packets to Connector on the uplink after adding metadata such as Signal-Noise-Ratio (SNR) and Received-Signal-Strength-Indicator (RSSI). Conversely, on the downlink, LoRa gateway simply performs transmission requests coming from Connector. It only transmits the received uplink/downlink packets as a relay without any interpretation of the payload. In addition, the packet forwarder project developed by Semtech is implemented on LoRa gateway, which defines the basic communication protocol between Lora gateway and Connector [16]. The packet types defined in the protocol are listed in Table I.

B. LoRa Network Server
LoRa network server is the important component for field deployment since it is responsible for data computing and packet processing in LoRa netwroks. Inevitably, the server desires for high capability to support the large number of LoRa nodes and then its architecture design is crucial. The porposed network server in X-LoRa consists of four modules for different tasks by means of microservice design. Moreover, a web project is developed in X-LoRa that makes it easy for administrators to manage LoRa nodes and gateway.
1) Connector: Connector is the bridge between the LoRa gateway and Central Server, which provides the services for parsing and packaging the physical layer payload based on LoRaWAN specification [17]. LoRa gateway can connect with a given Connector through a IP link and use User Datagram Protocol (UDP) for packet transmission. In addition, Connector performs legality verification on LoRa nodes and gateway to avoid unauthorized access. Due to the large number of LoRa gateway and nodes, Connector is required to parse and package millions of concurrent packets. In X-LoRa, more than just Connectors,the number of modules deployed can be flexibly adjusted according to different requirements. Thus, load balance is indispensable to evenly distribute the workload across more than one identical modules in order to improve the performace of the system and the utilization of all accessible resources.
2) Central Server: Central Server is responsible for data management and service scheduling. Firstly, Central Server invokes different modules according to the requirements of data processing. Depending on the type of uplink packet, the information in the packet is separated into the specific formats. Then, the original application data is fed into Application Server, the information about MAC commands is sent to Network Controller and the join packets are forwarded to Join Server without any interpretation. In addition, historical data is collected and stored by Central Server. It can provide the services for managers to check up the uplink/downlink packets and monitor the running states of LoRa node and gateway.
Owing to LoRa nodes are not associated with a specific LoRa gateway, the single packet is likely to be received by multiple LoRa gateways simultaneously. To avoid the waste of radio resources due to redundancy, Central Server is essential for filtering duplicate packets. Only one of the duplicate packets is fed into the subsequent processing modules. Moreover, the responsibility of Central Server includes scheduling of downlink packet transmissions. The transmission information such as RSSI and SNR attached in the duplicate packets is not discarded and then one of LoRa gateways is selected to send the downlink packet through exploiting this information. Finally, Central Server also determines the downlink parameters according to ralated configurations and reads the contents of downlink packets from two data queues which are responsible for application data and MAC commands.
3) Join Server: The services provided by Join Server include handling the activation requests of LoRa nodes and generating multiple communication keys. There are two methods to activate LoRa nodes, i.e., Over-The-Air Activation (OTAA) and Activation By Personalization (ABP). For OTAA method, the LoRa node must follow a join procedure consisting of the join request and the join accept exchange. To secure the activation process, the join request packet is only parsed by Join Server then the join accept packet is encapsulated according to the corresponding key. Furthermore, LoRa nodes using ABP method only need to be registered on the website through the specific management interface.
4) Network Controller: The functions of Network Controller focus on processing and managing MAC commands, which are exchanged between Network Controller and LoRa nodes in order to modify associated configurations or adjust transmission parameters [17]. In Network Controller, the welldesigned schemes are expected to be implemented to improve both reliability and energy efficiency for LoRa networks. Figure 3 gives the workflow of Network Controller in X-LoRa. Take Adaptive Data Rate (ADR) scheme as an example, the transmission parameters of the uplink packets are conveyed from Central Server to Network Controller. Then Network Controller performs the ADR scheme and determines whether to adjust either the data rate or transmit power. Finally, the MAC commands are added to a specific downlink queue and then are sent in the downlink packet. When receiving the reply acknowledgement packet, the successful commands are cleared from the queue. Otherwise, the failed commands are kept and wait for the next transmission.

C. Application Server
Application Server is responsible for handling, encryption and decryption of application payloads. It is necessary to support various applications with different encryption or encoding methods such as Protocol Buffer serialization to ensure data security and improve transmission efficiency. In addition, Application Server serves as a bridge between IoT cloud and the LoRa system so that customers can control LoRa nodes and enjoy the application data through web browsers or smartphones. The IoT cloud can operate via a topic-based publish-subscribe mode [13]. Therefore, the services provided by Application Server include publishing uplink messages and subscribing the specific topics to receive messages from IoT cloud.

IV. IMPLEMENTATION OF X-LORA
Aiming at supporting massive LoRa nodes in various IoT applications, X-LoRa is developed and its implementations are explained in this section.

A. Interaction between Modules
In order to establish reliable communication between models, Apache Kafka is used in X-LoRa as a publish-subscribe  messaging system [20]. The topic in the messagin system plays the role of the streaming data pipeline as the carrier of messaging. We can customize different topics for each module to achieve data exchange and service invocations between modules. As shown in Fig.4, each module subscribes one or more topics to fetch the corresponding data and publishes messages to their chosen topics.
There are a few advantages to adopt a messaging system to coordinate the interaction between modules. Firstly, the streaming data pipelines are acting as an abstract layer between consumer and producer which can reduce the coupling of modules and make systematic level clarity. As long as the message format is specified, the producer can be "ignorant" of the consumer. For example, Central Server only needs to publish the join-request message in the specific data structure to the topic JS-Pub subscribed by Join Server and know nothing about the implementation details. In addition, LoRa network server is also required to effectively solve the problem of high packet concurrency. The notion of streaming data pipeline inherits a plenty of features in the traditional message queuing model such as message order guarantee and asynchronous processing. Therefore, the messages are delivered sequentially to service providers and are handled asynchronously. Publishers do not need to wait for feedback which can avoid blocking caused by massive concurrent messages. This processing mechanism can meet the requirements of low latency and high performance in IoT applications.

B. Load balance
In X-LoRa, load balance across multiple identical modules is essential to support massive access and handle millions of concurrent messages. The consumer group is a collection of consumers which can run in separated processes or on separated physical machine. It is worth mentioning that each message published from the producer is assigned to only one consumer in the subscribing consumer group. The scheduler operates with the round-robin scheduling policy for all runnable consumers. Therefore, we can increase the number of processing modules in the same consumer group and load balance across them to improve the performance of entire system. Such method also can improve the fault tolerance of the network server. When a single module fails, the data needs to be processed by this module can be immediately switched to other parallel modules without service interruption.

C. Design on database
In order to meet the requirements of various data storage, Structured Query Language (SQL) and NoSQL database are utilized in X-LoRa. SQL databases are typically used to present data in the form of two-dimensional table with strict consistency of database transactions and persistent storage of data. However, the main bottleneck of SQL database is the inefficiency in handling high concurrent read and write which often happens in real-time IoT applications [19]. Therefore, NoSQL databases are utilized to provide real-time and highefficiency services for data storage.
The data needs to be stored can be divided into three categories in the network server. Firstly, the intrinsic information of users, LoRa gateway and LoRa nodes, such as username and unique identifier of nodes, is an essential component. This information requires persistent storage and has definite relationships and classifications. So we design a set of MySQL models for storing this category of data. In these models, data is separated into the logical and relational tables to avoid redundancy and to facilitate management. Besides, the historical data includes the uplink/downlink packets of each LoRa node and the status packets reported by LoRa gateway. This information has flexible formats and can be easily expressed by JSON format, which is similar to small documents. Therefore, a non-relational document-oriented database MongoDB is used in X-LoRa. Finally, some information needs to be temporarily stored during packet processing. For example, a Redis-based distributed lock is used to implement the uplink packet deduplication. Redis is an in-memory data structure store and has outstanding performance. The type of data does not require persistent storage but the input/output (I/O) speed of database need to be extremely high in order to meet the requirements of real-time data processing. In addition, a Redisbased caching mechanism is also developed to ensure the low latency and high performance of the network server when dealing with massive concurrent packets.

D. Deployment
All the modules of the proposed LoRa network server in X-LoRa are developed by Node.js. Due to a lot of I/O operations are processed on hard disk or memory in case of massive LoRa nodes, Node.js is based on an asynchronous I/O event-driven model that brings excellent performance and high load capacity with relatively low computing resource consumption [21]. It is worth mentioning that Node.js program runs in a single process while the physical machine or VM usually has more than one logical CPU. In order to make full use of hardware resources and guarantee the quality of the service, we can distributedly deploy multiple identical modules in separated processes or physical machine. Thanks to the scalability of the messaging system, the number of deployed modules can be flexibly adjusted according to requirements. In addition, the network server of X-LoRa also can be quickly deployed by Docker containerization [22]. We package all the modules in a  completed file system including project codes, databases and system libraries. Such method can simplify deployment steps and guarantee that each module runs correctly regardless of its environment.

V. EXPERIMENTAL RESULTS AND ANALYSIS
In order to evaluate the performance of X-LoRa, a LoRa network based on X-LoRa has been developed and deployed. The coverage performance of LoRa networks in a typical urban environment is evaluated in our experiments as well as the performances of the proposed LoRa network server with the specific metrics in this section.

A. Coverage performance and analysis
The LoRa gateway with wipe antenna is installed on the roof of a 15-floor building at the center of our campus. Then, the data packets collected through LoRa transmission are forwarded to the network server by LTE [10]. Both the LoRa gateway and nodes operate at the frequency band of 433 MHz. In order to evaluate the maximum distance of the LoRa transmission, LoRa nodes adopt the maximum spreading factor, i.e., SF=12, to transmit the packets. The main parameters in field trials are given in Table II. In this experiments, several LoRa nodes continuously send uplink packets and gradually move away from the location of LoRa gateway. The received signal quality parameters are measured about every 0.5 km. As shown in Fig.5, the RSSI and SNR decrease sharply within 1 km, while vary slowly with

B. Network server performance and analysis
In order to evaluate the network server performance of X-LoRa, all the proposed modules and the load testing program are deployed on two separate VMs. The main configurations of the VMs are shown in Table III. The testing program is developed based on Locust, which is a load testing tool and can support thousands of concurrent users on a single machine [23]. In our experiments, a LoRa node is treated as a user and each LoRa node generates an uplink application packet about every 40 seconds. We assume that all the packets are successfully transmitted by LoRa gateway so that the testing program can send/receive the uplink/downlink packets by UDP protocol to/from the network server in the specific format [16]. The number of LoRa nodes is set from 400 to 14,000 as deemed appropriate. Each uplink packet should be acknowledged by the network server and the response packet is received by LoRa node if the network server handles the packet successfully. Otherwise, if no response is received within five seconds, the processing of the uplink packet is regarded as a failure. When the number of LoRa nodes grows as expected, the throughput under the current situation is counted. Meanwhile, the median response time of each uplink packet is collected and average CPU utilization of Connector and Central Server is also used as a reference indicator. Our experiments are mainly focused on Connector and Central Server since both of them have significant impacts on the quality of the services for LoRa nodes under consideration. We also deployed the different number of Central Servers to demonstrate the changes in different metrics.
As shown in Fig.6, the throughput of X-LoRa is linear with the increasing number of LoRa nodes at the beginning since the network server can successfully process all packets. Then, the throughput becomes floor when the number of LoRa node reaches a specific limit, depending on the number of Central Server, e.g., 4000, 7,600, and 12,400 for 1, 2 and 4 Central Server, respectively. Meanwhile, the maximum throughput of X-LoRa is 98, 190 and 300 packets per second. It can be seen that an excessive number of received packages causes the network server unable to process the packages in time. Figure  7 shows that the median value of response time changes slowly when the number of nodes is less than the limit but increases sharply when the number becomes larger. Figure 8 shows the average CPU utilization of Central Server and Connector for different deployment condition. When deploying multiple Central Server, this curve shows the sum of the average CPU utilization of each Central Server. As can be seen in Fig.8 the CPU utilization of Central Server can slightly exceed 100% and 200% with 1 or 2 Central Server and the curves no longer rises with the increase of LoRa nodes. In addition, when the CPU utilization of Connector reaches 100%, the CPU utilization of Central Server also becomes floor. This is because the module is a single-threaded program developed by Node.js and then one Connector or one Central Server can only occupy one logical CPU. When any of the logic CPU no matter on which module is running is fully occupied, the X-LoRa system cannot accommodate more LoRa nodes. On the other hand, Central Server usually occupies much more computing resources than Connector so that Central Server is more likely to become a bottleneck. Therefore, deploying multiple identical modules simultaneously and achieving load balance between them can avoid such a problem so that the performances of the LoRa network server can be significantly improved.
VI. CONCLUSION X-LoRa has been proposed to provide a flexible solution for the implementation of private LoRa networks. The network server in X-LoRa consists of several decoupled modules with different functions, connected with streaming data pipelines to ensure the scalability and high performance. X-LoRa can help users to quickly deploy the private LoRa network and significantly reduce the cost and time for development of new LoRa applications. In addition, the project is open-source on GitHub to facilitate other developers and promote the further improvement. It is shown that the maximum transmission distance of X-LoRa is about 7.5 km in urban environments. Moreover, our experimental results demonstrated that the proposed network server can support more than 10,000 LoRa nodes with the reasonable computing resources.