Experimental Evaluation of End-to-End Delay in a Sigfox Network

This letter presents an experimental evaluation of end-to-end delay in Sigfox networks, considering measurements from the transmitter to the server. We carried out two types of cases. The first was a static experiment with a typical delivery time ranging between 2.5 and 4.5 seconds, with a 100% delivery rate. The second considers mobility, with the experiments carried out in mostly rural environments in which the transmitter is inside a car. In this case, we observed a delivery rate of 20% and an end-to-end delay of up to eight seconds. Our evaluation empirically indicates performance limitations that application developers should consider.

(which can be up to 12 bytes) per day and use an uplink data rate of 100 bits/s [5] in Europe with RC1. Messages are not acknowledged and are sent three times on different frequencies to maximize the probability that they will be received by the backbone network [6]. A downlink service allows four eightbyte messages per day; this technology can be classified as a low-power, wide area network (LPWAN). The structure of the uplink medium access control (MAC) frame is presented in Fig. 1. The size of each field in bytes is presented in parentheses in the figure. The maximum frame size is 26 bytes [2].
The payload of Sigfox messages can vary from 0 to 12 bytes for uplink messages and 0 to 8 bytes for downlink messages; however, the Sigfox protocol has predefined message sizes of 1, 4, 8, and 12 bytes for uplink messages and 8 bytes for downlink messages. The length is always one of these predefined lengths. If a given payload is between these lengths, protocol padding is added to reach a predefined length. For example, if the payload length is 6 bytes, then 2 bytes of padding will be added before sending [7].
In this letter, our focus is on reporting two types of experiments (static and mobile) for Sigfox networks in terms of end-to-end delay, a subject that has not been adequately investigated in the literature. Our contribution is to present our measurements to build a better understanding of the end-toend delays IoT application developers can expect when using Sigfox. The rest of this letter is organized as follows. Section II provides an overview of the literature. Section III introduces the static experimental setup used here, and Section IV presents the results for mobility. Section V concludes this letter.
We were not able to find any experimental-oriented papers that presented end-to-end delay measurements using the Sigfox network. Wang et al. [11]  technologies and reported delays from an end device to the Sigfox backend (gateway) to have an average of 3.63 s. Their results are not directly comparable to ours, as their study is conducted in the RC4 region, which has a data rate of 600 bit/s. Ribeiro et al. [8] examined mobility in rural areas. In their experiment, a Sigfox end device sent messages from a car moving on a motorway using RC2, which has a higher send rate of 600 bit/s and a higher transmission power of 24 dBm. They reported that 68% of the message transmissions were unsuccessful. In a study by Wang et al. [11], 73% of messages sent from a moving car were lost, but a moving drone at an altitude of 60 m was able to send all the messages successfully. Their experiments were conducted in RC4.

III. IMPACT OF MESSAGE SIZE ON DELAY MEASUREMENTS
We designed multiple sets of experiments to gain insight into how the Sigfox network behaves as a data channel. Our focus is on testing the end-to-end delays that IoT applications can encounter in practical use cases. The first set of experiments was conducted with fixed-size messages in a semi-urban environment. IoT messages were delivered to a Web and mail server, and their delays were measured. In the second set of experiments, the message size was varied to examine its impact, and in the third set, the messages were transmitted in a moving car. In these experiments, we used a Raspberry Pi (RPi) computer with Arduino MKR FOX 1200 to send Sigfox messages.

A. Delay Measurements With Fixed-Size Messages
The research setup was built in such a way that it could be run non attended for a long period of time. The research setup is illustrated in Fig. 2. The base of the setup is an RPi computer which runs the test sets. In these test sets, RPi make requests via serial (1) to the Arduino MKR FOX 1200 board, which has the capability to send messages to the SigFox network (2). The arduino board does not have an onboard real-time clock, so the use of RPi eases the experiment arrangements. Cronyd command is used for time synchronization in the RPi and Web server. For each send request, RPi records the timestamp, the message number and the time the send request took. This log information is later combined with other collected data. The Sigfox messages sent by Arduino are received by SigFox cell towers (and can be received by multiple cell towers) and forwarded (3) to SigFox middleware. The Sigfox network requires devices to be registered in order to use it, and through its middleware, messages from a specific device can invoke different rules for a customer. These rules can include message content rewriting to a human-readable format, addition of transmission data (signal strength, timestamp, etc.) to the message, and sending of the outcome to an email address or forwarding of the message to an Internet server. In this letter messages were forwarded both to a Web server (4) and sent to an email (5). Additionally, SigFox middleware was set to add a timestamp to each message.
The receiving Web server was implemented in a virtual Ubuntu 18.04 server run in a cloud by the IT Center for Science and located in Finland. The receiving Web server was implemented using Node.js, which recorded the time of receipt and the message content into a database. Google's Gmail service was used as the receiving email server. The email messages received were later downloaded, and the received message content and email timestamp were recorded in a database. In the experiments, information was collected in three places: a transmission log on the RPi, a reception on a Web server, and e-mail. The data were combined by transferring them to the same server and merging them into based on the message number. IoT messages received in e-mail were obtained by retrieving the entire e-mail box with Internet Message Access Protocol (IMAP), from which the script then parsed the IoT messages and arrival times into its own file and from which they were later combined with the data from other logs.
Trial arrangements were run for two weeks, and 2,087 messages were sent through the SigFox network during experiments. A message transmission interval of 10 minutes was used, which is the minimum transmission interval to comply with RC1 duty cycle requirements.
The results are documented as a dataset with a transit time from the sender, the middleware, the e-mail, and the Web server. Table I provides an example of the data collected, in which the first field is the message number, the second is the reception time in the Web server (Unix time, converted to milliseconds), the third is the e-mail reception timestamp, the fourth the timestamp on the IoT middleware server (MW), the fifth is the send time from RPi, and the sixth field is the calculated end-to-end delay from the RPi to Web server. The other metrics, such as the Sigfox  A more detailed study included 1,553 messages on the ground that some server issues were excluded from the study. Based on the analysis, the minimum time it took from the start of sending a message to receiving it in the Web server was 2.396 s and the maximum was 19.944 s. All messages sent  TABLE II  END-TO-END TRANSMISSION DELAY FROM RPI TO A WEB SERVER   TABLE III  TRANSMISSION DELAY FROM THE SENDER TO THE MIDDLEWARE were successfully received. The distribution of message transit times is shown in Table II, in which it is shown that 26% of the messages were received in less than 2.5 s and 62% of messages in less than 3 s. In about one percent of the cases, it took more than 6 s to receive the message.
The time elapsed from the transmission command given by RPi to the time receipt reported by the SigFox middleware was calculated and the distribution of message transit times is shown in Table III. The middleware reports the time receipt of a message with an accuracy of only 1 s, which limits the accuracy of the study. In this case, 98 percent of the messages were received in less than 3 s.
The test application was sending 5-byte messages, which were then padded to 8 bytes and packed within Sigfox message protocol. The maximum size of the Sigfox frame is 26 bytes. If we assume that the frame header has a fixed size and we have 8 bytes of data to transmit then we will obtain an estimate for the time it takes to send the frame over the air. The transmission time equals (header size in bits + payload in bits) / 100 bits/s, which gives us (14*8 + 5*8 + 3*8) / 100 = 1.76 s aligned with the 1.75 s stated in [7].
It should be further noted that we do not have information on the synchronization of middleware clocks, and the time resolution provided by the middleware is 1 s, which causes uncertainty in the results. From these findings, it can be concluded that messaging from the middleware to the server has resulted in longer delays for several messages. The reason these may be the congestion of Internet traffic between The time which it takes to receive an IoT message in an email server is also studied because in many cases, IoT notifications can be sent as email notifications. The setup has many variables, which are beyond our control, like 1 s email receipt timestamp resolution, so readers should treat these results as an example of a case study. The distribution of message transmission times from the send command in RPi to the receipt of the message in an email server is presented in Table IV. According to the results, 87% of the messages arrived on the email server in less than 4 s.

B. Impact of Message Size
We designed a set of experiments to evaluate the impact of message size on the transmission time, connection quality and connection reliability using a similar setup as in the earlier experiments. This setup is illustrated in Fig. 2, but this time the sending of messages to a mail server was not included. Similar to the first experiment the transmission log was kept on the RPi, which was later combined with the information from the backend server. Messages were sent with a payload sizes of 1, 2, 4, 8, and 12 bytes. When a 1-byte message was sent an unsigned integer was used as the message payload. For all other message sizes, the message payload contained the message number. For these tests, we upgraded our Ubuntu server to 20.04 LTS in the IT Center for Science's cPouta cloud hosting service. The receiving server was implemented using Node.js and MongoDB Atlas for the storage of message data. A total of 562 messages were sent during the experiment.
The average transmission times were evaluated from the messages sent to the cPouta backend. The average endto-end transmission times and the minimum and maximum transmission times are illustrated in Fig. 3. For the average transmission times the standard deviation is drawn as error bars in the figure. For example, for 1-byte messages, the standard deviation was 427 ms, which is marked in both the positive and negative directions in the chart. On average, 66-78% of  To illustrate the distribution of the message transmission times, a cumulative distribution function (CDF) chart of the transmission times was drawn. The chart is shown in Fig. 4, indicating, for example, that at 3 s, about 40% of 1-byte messages have arrived. This plot is drawn based on the actual observed end-to-end transmission times. The connection reliability in the experiment was found to be excellent, as all sent the messages arrived successfully at the backend server. Out of all messages sent, 89.6% were received by three or four base stations. On average, the messages were received by 3.04 base stations. The average Received Signal Strength Indication (RSSI) value between base stations was found to be about −124 dBm.

IV. MOBILITY EXPERIMENTS
As IoT devices are extensively used in logistics, better understand of the performance of Sigfox technology in mobile setups is also important. Motivated by this, we have carried out a series of tests in which we placed our measurement system in a car to determine how mobility impacts message delivery times, connection quality, and reliability. During the experiments the driven route was recorded with a separate Global Positioning System (GPS) application with a 1 s resolution. Route information was combined with the time the messages were sent using timestamps, so we obtained the GPS coordinates of the transmission locations and calculated the  The collected position data were visualized with Qgis software [22], which enabled us to tag transmission positions based on transmission time. Fig. 6 shows an experiment in where a car has traveled 260 km, and our system attempted to send 19 messages (one message every ten minutes). In the figure, successful transmissions are indicated with a green marker. The estimated vehicle speed is marked for every attempted transmission.
Three mobility experiments were conducted for a total of 518 km traveled. The routes mostly included rural areas in Southern Finland. These involved a large portion of motorways and other bigger roads. The average speed of the vehicle in the experiments was 80.3 km/h.
In the experiments, a total of 39 message transmissions were attempted, eight which were successful, with the delivery times varying from 3.47 s to 7.89 s. This means that in the experiments, 20.5% of the transmissions were successful. It was also observed that three of the eight successfully received messages had noticeably higher transmission times than the average transmission time observed in the static experiments. It was assumed that Sigfox sends messages three times on separate frequencies, and the first or second transmission is not received by Sigfox network, accounting for the observed result. All the three messages with higher transmission times were received by a single base station. The delay distribution of the experiments is presented in Table V. The studies  TABLE V  END-TO-END TRANSMISSION DELAY IN MOBILITY EXPERIMENTS by [8], [11] also reported a significant number of lost messages in mobile experiments, which is consistent with our results.

V. CONCLUSION
In this letter we analyzed the Sigfox network as a messaging channel for applications. In the stationary experiments, messages were delivered reliably. The typical message delivery times from a sending device to a server in the Internet varied from 2.5 s to 4.5 s depending on the message size. However, mobility tests revealed that the message delivery time could be notably longer in non-optimal situations. We assumed that this was due to the design, in which the message is sent three times on different frequencies, and to whether earlier transmission is lost and later received, causing an increase in end-to-end delay.
In summary, we list what we learned from our experiments in the following: • The typical end-to-end delay, with a static sender and a server in Internet in semi-urban environment with 8-byte message, is less than 3.5 s. Our application server in the Internet received 26% of the messages in less than 2.5 s, 62% in less than 3 s, and 88% in less than 3.5 s. However, there were cases in which delivery took notably more time, but these were assumed to be caused by the general nature of Internet traffic. • Message size affects the message transmission time. Each message is padded to a size of 1, 4, 8, or 12 bytes, which the developer should be aware of. In our experiments, message size affected the average end-to-end delay to a server in the Internet with respect to message size of 3.1 s, 3.5 s, 3.8 s and 4.1 s. • Mobility can have a drastic impact on message delivery reliability, as only 20% of the messages were delivered successfully in our experiments. However, the results could be different if the antenna was located on the roof or if the experiments were run in an urban environment. • The user cannot expect that the message is delivered reliably. The Sigfox network can provide four downlink messages per day, so acknowledging the messages is generally not an option.