Evaluating Indoor Positioning Systems in a Shopping Mall: The Lessons Learned From the IPIN 2018 Competition

The Indoor Positioning and Indoor Navigation (IPIN) conference holds an annual competition in which indoor localization systems from different research groups worldwide are evaluated empirically. The objective of this competition is to establish a systematic evaluation methodology with rigorous metrics both for real-time (on-site) and post-processing (off-site) situations, in a realistic environment unfamiliar to the prototype developers. For the IPIN 2018 conference, this competition was held on September 22nd, 2018, in Atlantis, a large shopping mall in Nantes (France). Four competition tracks (two on-site and two off-site) were designed. They consisted of several 1 km routes traversing several floors of the mall. Along these paths, 180 points were topographically surveyed with a 10 cm accuracy, to serve as ground truth landmarks, combining theodolite measurements, differential global navigation satellite system (GNSS) and 3D scanner systems. 34 teams effectively competed. The accuracy score corresponds to the third quartile (75th percentile) of an error metric that combines the horizontal positioning error and the floor detection. The best results for the on-site tracks showed an accuracy score of 11.70 m (Track 1) and 5.50 m (Track 2), while the best results for the off-site tracks showed an accuracy score of 0.90 m (Track 3) and 1.30 m (Track 4). These results showed that it is possible to obtain high accuracy indoor positioning solutions in large, realistic environments using wearable light-weight sensors without deploying any beacon. This paper describes the organization work of the tracks, analyzes the methodology used to quantify the results, reviews the lessons learned from the competition and discusses its future.


I. INTRODUCTION
T HE international conference on Indoor Positioning In- door Navigation (IPIN) brings together every year around 400 experts from the industrial and academic world to discuss the uses of indoor geolocation, discover the best technologies in indoor positioning and indoor navigation and share the latest innovations.The 9 th edition took place in Nantes from 24 to 27 September 2018 and reunited more than 40 nationalities.A competition took place in parallel to the congress and allowed teams to compete against each other on pedestrian tracks in a very challenging realistic scenario: a shopping mall.
Several previous indoor competition editions were held within the EvAAL framework [1], [2] in the past.Although the IPIN Conference runs competitions in parallel to the conference using the EvAAL framework since 2014, this approach was formally adopted by the IPIN Competition in 2016 [3].Other competitions, such as the Microsoft Indoor Localization Competition at the International Conference on Information Processing in Sensor Networks [4]- [6] or the GEO-IOT world [7], targeted similar on-site competitions.The main goal of the on-site competitions is not only to assess the performance of the competing systems but also to evaluate the ability of the teams to configure their indoor positioning solutions and to successfully work in a new environment in a very short time, usually in less than 24 hours.In the past, on-site competitions were mainly performed in conference venues, including hotels, convention centres and university buildings.
Evaluating indoor positioning systems through data sets is now emerging (e.g., Perfloc [8] or the IEEE CTW 2019 Positioning Algorithm Competition [9]), since it enables all teams to work with the same data and technologies for fair bench-marking.Contrary to on-site competitions, where the teams have to survey the area in their own way for calibration purposes, off-site competition organizers provide all the required data to calibrate and evaluate the indoor positioning systems.Thus, since IPIN 2015, the indoor competition includes an off-site track.At first, it was based on a Wi-Fi fingerprinting database collected at three different buildings in a university campus [10].Since 2016, the Smartphonebased off-site track provides all data that can be captured in real-time with a smartphone [11]- [13].Using data sets for the competition has enabled the inclusion of new buildings, which in turn added diversity to the evaluation areas.
However, these past competitions did not include complex realistic indoor spaces such as public populated environments, which provide an extra challenge and are of high relevance for any real-life indoor application.Although the conference venues were populated, some restrictions applied to the on-site competitions to avoid disturbing the conference regular activities.Therefore, in 2018, we decided to target this issue and organized the IPIN competition in a shopping mall, Atlantis.Thus, calibration, site surveying and evaluation were performed under realistic conditions during commercial opening hours, including rush hours with crowded corridors.To better reflect the challenges introduced by this new scenario, as well as the latest innovations in indoor positioning technologies, a total of four different competition tracks, two on-site and two off-site, were organized in the shopping mall.This paper aims at presenting the methodology adopted to organize this competition and discussing the results achieved by the competitors before drawing general conclusions.
The paper is structured as follows.Sections II and III describe the preparatory work: design of the pedestrian routes, geo-positioning of the reference points in the building and creation of data sets for the off-site tracks.Section IV focuses 2 VOLUME 4, 2016 on the different approaches deployed by the competitors.Section VI draws the lessons learned from the competition at the levels of the organization and the participants.

II. PREPARATION OF IPIN 2018 COMPETITION TRACKS AND DATASETS
This section describes the preparation of the IPIN 2018 competition tracks, which has been carried in partnership with the French High School of Surveyors and Topographers (ESGT), under the supervision of the GEOLOC laboratory of the French Institute of Science and Technology for Transport, Development and Networks (IFSTTAR).The competition was also prepared in collaboration with the Institute of New Imaging Technologies (INIT) from Universitat Jaume I and other institutions.
The IPIN competition basically consists of two main parts: an on-site and off-site competition, although every part can have one or more tracks in the competition: • The on-site challenges occur at the competition location, i.e. the conference venue or, as in this case, a shopping mall.The teams deploy their own Indoor Positioning System (IPS), respecting and obeying the restrictions set by organizers, and estimate in real-time the coordinates of the target points as they are reached by the person carrying the location system and following a track previously arranged by the organizers.Competitors are allowed to access the venue the day before the competition for calibration and site survey.• The off-site challenges allow competitors to calculate traces from data provided to them before the competition.Data are of various nature: Wi-Fi, magnetic field, inertial signals, Bluetooth Low Energy (BLE), etc.They comprise two parts provided to competitors: The first part of the dataset is recorded on a geolocated path (or paths) where the geographical coordinates of the reference points and the corresponding data are provided; this path is used for calibration; the second part of the dataset are raw data from the same location where previous data were taken, and their coordinates must be estimated by the competitors.To ensure fair competition, an additional site survey is not allowed to competitors.In both cases, a common evaluation framework is applied.The individual positioning error corresponds to the 2D positioning error plus a penalty term in the case of incorrect floor detection.The final score is either calculated in meters at the third quartile or extracted from the function of cumulative error distribution at the third quartile.This information corresponds to the highest error for the 75% of the estimates.Several recommendations of the ISO/IEC standard 18305:2016 [14] have been taken into account to define the IPIN 2018 competition performance metrics.

A. TWO ON-SITE CHALLENGES
On-site challenges involve competitors who compute the coordinates of the waypoints on which they walk, in real-time and using their technology.Computation was not limited to the embedded or hand-held devices, but competitors had to provide the estimated locations of the evaluation waypoints immediately after reaching the end of the evaluation path.i.e., post-processing was not allowed.All estimated coordinates must be reported by the competitors to the organizers when the end of the route is reached.On-site competition comprised two challenges, which only allowed hardware worn on the upper body part, prohibiting Lidar technology and infrastructure deployment in the environment.Indeed, Lidar mobile mapping systems are generally for accurate indoor mapping.They are considered as reference solutions rather than personal navigation systems to be tested.As described in the next section, a Lidar system was used to compute the 10 cm accurate coordinates of all the waypoints.There were two on-site competition tracks: camera-based and non-camerabased.

1) Track 1: Camera-based positioning challenge
The 1 st on-site challenge allowed the use of camera enabling vision-based localization.All vision-based technologies were accepted except laser-based technology, like Lidar, that was excluded from the competition.Competitors were allowed to visit the shopping mall during normal opening hours the day before the competition for calibration and surveying purposes, but they were not allowed to install any kind of hardware that could help their device to estimate the location (e.g., BLE beacons, Wi-Fi routers or UWB tags).This prohibition was also in compliance with the Infrastructure-Free requirement placed by the shopping mall regulation.The track organizers provided the coordinates and heading of the starting point on the competition day.During the competition, competitors were asked to perform some specific actions, classically done by pedestrians indoors.Fig. 1 illustrates some of these actions: sitting on a bench, going upstairs and standing on a conveyor belt.

2) Track 2: Non-camera-based positioning challenge
The 2 nd on-site challenge followed the same rules and requirements as Track 1, but the use of cameras was forbidden.All equipment had to be carried on the upper part of the human body, including trouser pockets.This restriction was added to the track to explicitly exclude foot-mounted Pedestrian Dead Reckoning (PDR) techniques, addressed in Track 4. Competitors were asked to perform similar actions as in Track 1.

B. TWO OFF-SITE CHALLENGES
Following a pre-study and the experience of previous IPIN competitions, two off-site tracks were defined for the off-site challenges, namely the Smartphone-based positioning challenge and Foot-mounted IMU and GNSS based positioning challenge.VOLUME 4, 2016 The first off-site challenge aimed at testing the location technologies solely based on smartphone measurements recorded by a person who had walked through the mall.This track matched the off-site tracks organized in the 2016 [12] and 2017 [11] IPIN competitions.
To determine the parts of the shopping mall that were usable for the off-site smartphone-based positioning, several data acquisitions were carried out with the "GetSensor-Data" Android application version 2.0 [13].This application records all data acquired with most of the sensors embedded in a smartphone (WiFi RSS, Inertial measurements, Bluetooth low energy RSS, GNSS location, pressure, light and sound intensity,...), and stores it in a log file.We divided the shopping mall area into five parts and performed a loop in each part to collect the data.These data were analyzed to identify the areas that could not be used for the competition: without Wi-Fi or BLE coverage or white coverage areas.The analysis showed that all regions were sufficiently covered, except for some parts of the car parking areas where the Wi-Fi signal was particularly weak.Data provided to the competitors for this challenge were registered with the same "GetSensorData" Android application in three phases: training, validation and evaluation.
For the training and validation phases, some short trajectories were defined in the evaluation area to record the required data.Training data is usually used for calibrating the IPS, whereas validation data can be used by the teams to estimate the accuracy of the developed solution.However, competitors can use both datasets at their own discretion.Each short trajectory was collected multiple times in both walking directions.A total of 22 training and 15 validation log files were provided to the competitors.
A 1 km path was recorded with the application for the evaluation phase, generating a blind evaluation log file without reference waypoints.This dataset provided to the competitors for track 3 [15] (raw signals and reference points coordinates) can be downloaded for further bench-marking.Fig. 2 (left part) shows the data acquisition for Track 3.

2) Track 4: Foot-mounted IMU and GNSS based challenge
The second off-site track was dedicated to foot-mounted inertial and GNSS navigation.The data were collected with the unit PERSY (Personal Reference SYstem, Fig. 2 right part) developed by GEOLOC at IFSTTAR.It comprises a 9 degrees of freedom inertial unit and a GNSS receiver and antenna [16], [17].Two datasets were prepared.The first one was used for sensor calibration and the second one for the calculation of the coordinates of the waypoints.The usage of maps was completely forbidden in this track.
First, the magnetometer calibration dataset was recorded in an undisturbed geomagnetic environment by performing rotations around the different sensor axes.It is a standard procedure for ellipsoid fitting based magnetometer calibration algorithms [18].It was complemented with data for conducting an Allan variance analysis [19], [20] of signals recorded during several hours in a static location to determine noise and measurement bias of inertial sensors.These data were provided to the participants to enable them to calibrate the device with their algorithms.
Finally, a dataset was recorded over 1 km of walking distance on three floors and provided to the competitors for the computation of the coordinates of the waypoint.The purpose of this challenge was to compare pedestrian navigation algorithms based on inertial, GNSS and magnetic measurements on similar data.The dataset provided to the competitors for Track 4 [21] (raw signals and reference points coordinates) can be downloaded for further bench-marking.

III. ACCURATE MAPPING OF THE TRACKS
This section describes all the steps carried out to accurately map all the tracks, in the form of a sequence of ground-truth points whose coordinates were determined with 10 cm accuracy to enable the competitors' IPS performance assessment.

A. DEFINITION OF THE EVALUATION TRACKS
The first task consisted in defining the evaluation tracks in the shopping mall for the four challenges proposed in the IPIN 2018 Competition.To avoid that competitors in one Track could get information from other tracks, different paths were used for every single challenge.On the other hand, it is important to note that it was the first time that, to make it more realistic, the competition took place in a very large, real-life public environment, which complicated the challenge.The shopping center Atlantis has 151 stores and 31 restaurants on a 14 000 m 2 area spread over 3 levels.
The four paths were sketched on the floor maps.We considered the positioning technologies involved in the challenge when designing the evaluation track and included some challenging parts such as stair climbing, passages by elevator or on a treadmill, breaks on benches, and other real-life activities in the shopping mall to resemble realistic walking trips.Table 1 shows the description of the four paths.
Finally, the paths were plotted on the map and then characterized as a sequence of representative waypoints, mostly turns and intermediate points in long rectilinear paths.A total of 180 waypoints were identified on the map.All the waypoints were later surveyed and mapped into WGS84, serving as ground truth.Although some waypoints are shared among the four tracks, their sequencing and the actual pedestrian route needed to reach them is different.Evaluation tracks with a large number of evaluation waypoints are necessary to conduct relevant statistical analysis of the positioning errors.

B. MATERIALIZATION OF THE TRACKS
The different paths were mapped in Autocad using the building's floorplan, after a stage of clean-up and adaptation to our purposes.The organizers used small and discreet self-adhesive pads (Fig. 3) to mark the waypoints located on the ground surface.The solution had to be easily and quickly positioned on-site (due to their large number), not very visible (so as not to disturb the shopping mall staff and attract the attention of customers) and should be able to last for the surveying over several days.We placed the 180 stickers throughout the gallery, using the Autocad plans previously made.For each sticker, we had prepared an identification card with the serial number, sketches, photos and explanatory notes.These cards were used on the competition day to reposition all the waypoints.

C. SETTING UP A NETWORK OF REFERENCE SURVEY POINTS
Given the international nature of the competition, the geographical coordinates used in the competition were expressed in the standard system WGS84 instead of using a local or regional projection system.The coordinates of the waypoints were therefore provided in WGS84.Other geographical systems were used to post-process the measurements recorded by two dynamic scanners.The French RGF93 CC-47 projection system was adopted and the deformations due to linear distortion were compensated in the estimation of the coordinates.
As no reference points were marked in the various plans provided by the building's architect, RGF93 reference points were surveyed.Two GNSS receivers Leica System 500 were used using the free pivot method.Five reference points were materialized and accurately surveyed outdoors.Several GNSS points, previously surveyed on the roof (Fig. 4), were used to ensure a good distribution throughout the shopping mall.The network of reference points is shown in Fig. 5.
Thanks to these reference points, it was possible to compute the coordinates of two stations located in front of the shopping mall entrances.Indoor stations were determined by theodolite measurements and served as reference points for the 3D scanner indoor mobile mapping.All indoor surveyed coordinates were transformed from RGF93 into WGS84 using the EUREF Permanent Network website considering that RGF93 corresponds to ETRF2000 and that ITRF2014 version 2017.91 is homogeneous with WGS84 within 10 cen- timeters.The same system was kept for the competition day that went through several months after the mobile mapping.

D. MOBILE MAPPING OF THE SHOPPING MALL
A total area of 9000 m 2 was dedicated to the competition and surveyed by two mobile scanners deployed by the Viametris company [22].The bMS3D is carried in a backpack whereas the iMS3D is a mobile robot scanner.Both systems are shown in Fig. 6.
The iMS3D was used in the galleries on the ground floor and the bMS3D was used for the paths on different floors, stairs and pedestrian parts in the car park area.The survey lasted two hours and did not include GNSS data.The bMS3D scanner surveys points up to a maximum distance of 30 meters with minimized noise but can theoretically aim for further distances.The iMS3D scanner surveys points up to 30 meters, but it is programmed to 10 m to 15 m for the same reasons.The iMS3D includes three profiling lasers and a panoramic camera to colorize the point cloud.Unlike the fixed laser scanner, which measures all around it thanks to a rotation of its laser, the profiling laser propagates only in lateral and horizontal planes.It is the movement of the operator that allows the acquisition of a complete point cloud.Its characteristics are a precision of 2 cm to 3 cm for a resolution of 80 000 points/s.It has a 10 cm per 100 m deviation in the horizontal plane.Coordinates were estimated by Simultaneous Localization And Mapping (SLAM) processing.[23], [24] give tutorials on SLAM.The bBMS3D comprises four cameras and two scanners.The cameras take pictures every second to colorize the cloud.An inertial unit estimates roll and pitch angles and provides the horizontal plane reference.Its characteristics are 50 cm per 100 m 3D deviation and 1 per 1000 deviation for the X and Y components.50 cm up to 1 m deviations were observed in the dataset.
Known coordinates on the corners of walls at the ends of the gallery were needed for the global point clouds adjustment in a post-processing mode with PIMMS software [25].They were surveyed with a theodolite and used to georeference the SLAM acquisition.The backpack survey started on a known reference point allowing connecting with iMS3D point cloud.A 5 per 1000 deviation was observed on the global adjustment due to dynamic obstacles (movements of persons) and due to the reflections on the mirrors and the windows of shops through the gallery.The start and end positions of the acquisition were identical to close the loop and bond the drifting errors.Finally, the point clouds were colorized using the photos to produce more realistic restitution of the scenes.Fig. 7 shows mobile mapping outputs and [26] is a video of the results.It was found that even the tile joints and the writings on the ground stickers could be seen in the final point clouds.The targets' coordinates were then extracted by hand from the georeferenced points clouds by analyzing the images.A 2 cm to 10 cm standard deviation on the 3D positioning error was obtained for all 180 reference points used to define the competition tracks and assess the performances of the competitors' systems.Maps were provided to the competitors -except to the teams participating in Track 4-as supplementary materials.
Team Ariel 1 considers a navigation system that includes a visual sensor (camera) and an orientation sensor.The process of extracting visual landmarks consists of the following two steps: Image processing and Extraction: Image Processing: the ceiling lights are considered as landmarks, and a simple threshold filter can be used for their extraction as depicted in Fig. 8.That results in a binary image I b from which the center mass, c, is extracted of each visual landmark.The process is performed in two main steps: first extracting the contour γ of each landmark that appears in I b ; and second, average through all points g ∈ γ.Fig. 9 shows the general concept of using lights for indoor navigation.Few additional geometric properties of each light-source are analyzed, including center, radius and a compact representation of its perimeter (contour).Extraction: For each spotted visual landmark center c, represented as a two-dimensional vector c = (c x , c y ), the relative world vector connecting the visual sensor and the landmark (in world coordinates) is pursued.The first step is to calculate the Intrinsic matrix K 3×3 of the visual sensor which is known as "Camera Calibration".The relative vector is then: where w is a scale factor of mapping R 2 to R 3 which is unknown.Hence, v ∈ R 3 can be calculated up to a scale.Finally, the acquired vector c is rotated by the device selforientation to align it with the world coordinate system.
b: Put It All Together The system used implements pedometry (i.e step-counter, optical flow), that is used as a rough position estimation to define a region-of-interest.The fusion problem is addressed with a modified particle filter.Commonly a particle filter is used with the initial area of the filter covering all the  evaluation map (and converging to the most likely Probability Density Function).Instead, in the proposed system the search is confined to the Region Of Interest (ROI).Once the ROI is reported the relevant particle's weight is increased.This way, the true position is likely to be inside the predefined area.To fuse detected visual landmarks, each particle's weight should be modified.Weights are modified according to some weight function that compares the particle state with the true state.Some particles are periodically spread outside the ROI to overcome the 'kidnap robot' situation.The generic particle filter algorithm is presented below: A real-world localization scenario often includes complex sensory data: outliers, inaccurate and partial sensory information.Moreover, human factors can influence the sensor measurements and therefore contradict the pure Bayesian inference filter, see [27] for details regarding the ways to overcome those challenges.

c: Visual Landmarks Mapping
The process of visual landmark mapping follows the following steps: 1) preprocess the visual measured data and extract the visual landmark vectors v w that correspond to each landmark (Section IV-A1a); 2) the visual landmarks are then registered (mapped) in real-world coordinates relative to the environmental map (note that the task of mapping a complex building such as a shopping mall that may have hundreds of light sources might be challenging and timeconsuming); and 3), a light source is spotted and tracked along a path, the position of the light source is then estimated as the weighted average of all the intersections of the 3D vectors to that source.d: Field Experiment and Result in a shopping mall Few large scale experiments were performed in crowded shopping malls.Each experiment was divided into two steps: 1) Mapping; and 2) Localization and accuracy evaluation.Mapping an average size shopping mall can be done within a few hours including testing the map and validating it.The level of accuracy varied between high accuracy (0.5-2 meters) and low accuracy (5-20 meters) when the visual sensor was blocked.

2) ARIEL2 Team
Team ARIEL 2 presents a smartphone indoor positioning system (IPS) based on recent Augmented and Mix Reality (AR and M R) tools such as Google's ARCore or Apple's ARKit.The AR tools are used as visual pedometry (scaled optical flow) sensor, which is then fused with an advanced version of a localization particle filter to produce both accurate and robust solutions for various indoor positioning applications.The presented method allows a simple and efficient mapping solution that, combined with the localization particle filter, allows 1-2 meter positioning accuracy in most standard indoor scenarios.

a: The basis of Indoor Position
The user global position can be retrieved from existing geolocation services (e.g., Google Maps Geolocation API).Such user location is commonly approximated using radiofrequency (RF) signals (4G-3G, WLAN, BLE) and even weak global navigation satellite system (GNSS).The accuracy of such methods is considered to be "building level" (10 m to 50 m) or "room level" (5 m to 10 m).The user's relative position is often computed using a pedometer.The smartphone-based pedometer is composed of two major virtual-sensors: 1) "Step-counter": which detects discrete step-events; and 2), Orientation sensor, which approximates the user's global/relative direction.Combining the two parts allows a step-based relative path computation.Naturally, such a method tends to drift in time (and steps).Many modern IPSs combine the above two positioning methods to achieve an accurate and global localization (with no drift).

b: Particle Filter for Localization
A particle filter method represents the posterior distribution of a set of particles P (|P | = n) on a given map, the result of such algorithm is a new set of particles P with a (slightly) different distribution (see Algorithm 2).
Input: Black and white 2D map of each floor in the navigation region.Init: generate a set P of n particles.For every x i ∈ P a random location < x, y, z > in uniform distribution over the map.Result: Estimated 3D-Position : l =< x, y, z, > while True do 1) Estimated the current movement < ∆x, ∆y, ∆z >.
2) Apply an action-function on all particles in P .
3) Evaluate the weight of each particle according to its new position on the map.4) Re-sample the particles → P 5) Report the current best-position.end Algorithm 2: Generic particle filter localization algorithm In order to implement the above generic algorithm the following functions should be well-defined.
• Movement estimation: computes the 3D movement vector according to the scaled optical-flow algorithm.• Action function: Translate each particle according to the movement vector (in 3D).To prevent the convergence of the particles from happening too fast (and by that risk missing the true location), a small random noise is added to each particle.• Evaluate: compute the updated weight of each particle according to the map constraints.The third feature is exploited to improve the mapping and The advanced particle filter relies on the existence of a pre-made map (of the region of interest) divided into the following "colors", shown in Fig. 10.A: Accessible area (white), B: Unaccessible area, such as walls and fixed barriers (black or brown), C: Accessible area, in fixed width, tied to the B regions (pink), D: Area near floor-changing, such as stairs and elevators (yellow).The generated map is the foundation of the particle filter algorithm and further used to determine the particles' weights.3) ETRI/HANA MICRON Team ETRI/HANA MICRON team (ETRI team), as a part of an Indoor Navigation project, investigate vision-based localization approaches.They are interested in monocular visual SLAM applications in public buildings like airports, train and subway stations, shopping malls, etc.A common problem in such dynamic environments is the presence of moving people and objects in the input images which degrade the accuracy of the localization results.
To deal with this, ETRI Team focused on the use of semantic information to process detected sparse features according to their reliability for localization purposes.A monocular SLAM is developed, following a common approach [28] using sparse feature-based visual odometry in combination with semantic segmentation filters on the input images.
In parallel to this approach, which requires continuous streaming of images from the phone camera to the PC based implementation, single image-based localization approaches with much lighter communication load were also investigated.More specifically, PoseNet [29] (based on the PyTorch implementation [30]) for relocalization was experimented, planning to replace it later with the map-based relocalization provided by visual SLAM.
The main implementation was not completed in time for the IPIN2018 competition and was ended up using the PoseNet based relocalization with a filter over the estimated positions.The implementation ran on a notebook computer receiving images from a smartphone at about 5 frames per second.On the day before the competition, images were collected from reference points (spaced about three meters apart) along a set of likely trajectories in the mall.The dataset preparation and model training were performed using cloud servers.
Looking back at the results, the relocalization approach performance was affected by the sparseness of the collected dataset and the limited time for model training/tuning.For example, several sections of the testing trajectory were outside of the areas covered by the dataset.Collecting only once along a trajectory resulted in the inclusion of a bright red service vehicle (used in light fixtures repair) that was gone by the time of the test run.In cases like this, when sufficient variance in the scene could not be captured, the ability to use additional semantic information would have been helpful.

4) GOOGLE Team a: AR Motion Tracking
Recent developments on ARCore [31] for Android devices (based on the previous Project Tango) and ARKit [32] for iOS made it possible to accurately track motion using visualinertial odometry (VIO) [33] in smartphones.The types of phones with ARCore enabled include Google Pixel, as well as various Samsung, LG, and Asus phones.However, the tracking is estimated with respect to the initial camera pose, not in a global coordinate frame.The Google Team proposed a method to automatically align a local trajectory with global coordinates, using either a few accurate landmarks -or reference positions -, such as visual markers on the map, and/or many noisy landmarks, such as the ones from the Android Fused Location Provider (FLP) [34].

b: Rigid Trajectory Alignment
The solution proposed is a least-square minimization between global locations (λ, φ) where λ and φ stand for longitude and latitude in radians, and (x, y) are the local coordinates with respect to the initial pose of the camera, subject to the rigid 2D transformation: (x 0 , y 0 , α).Timestamped local trajectories are recorded with corresponding global anchor locations, then perform point-wise least-square error minimization is performed: with w a weight for the position.Anchor (x b , y b ) corresponds to the locations from global latitude / longitude: where R is the radius of the Earth (6 378 137 m) and (λ 0 , φ 0 ) indicates the start location of the camera, (x t , y t ) are the local trajectory coordinates obtained by motion tracking: For FLP locations, w was set to 1/d 2 where d is the reported accuracy in meters.For user-provided accurate landmarks, d was small, e.g.0.1 m, which results in large w.Given the 2D transformation (x 0 , y 0 , α) computed from Eq. ( 2), and any coordinates from the location trajectory (x t , y t ), the locations in the global coordinate system are obtained: The optimization is performed at a 0.1 Hz fixed sampling rate in real-time.

c: Landmarks from Initial Location and Direction
For the 2018 IPIN competition, the initial location (λ 0 , φ 0 ) and direction (θ 0 ) were given and used as accurate global markers.The initial location (λ 0 , φ 0 ) is a landmark by definition.The initial direction was used to compute the second landmark, by where L is the length along the initial direction.

d: Floor Detection
The accuracy of ARCore is around 2 cm in the horizontal plane, but it is noisy vertically.Vertical changes obtained from ARCore for indicating the vertical movement only, and the barometer sensor was used to obtain the actual height changes.It was assumed that all floors were the same height.
The system worked well in most buildings, but failed in Atlantis Mall in testing, as the height of the second floor was significantly lower than the first.

e: Multi-segment Alignment
Although ARCore is accurate locally, it may jump in distance or direction from time to time.It is mandatory to detect those jumps and perform global alignment to each segment.All segments except the first are used as FLP global anchors, as no interaction with the map can be performed during the test.

B. TRACK 2: NON-CAMERA BASED POSITIONING CHALLENGE 1) ETRI/SNU/HANA MICRON Team
This technical description presents the indoor positioning technology that was implemented on a smartphone for the Track 2 of IPIN 2018 competition by ETRI team.Most smartphone-based indoor positioning applications implement a hybrid approach based on multiple resources such as Wi-Fi, BLE signals and inertial sensors.Since the proposed ongoing project targets commercially available product, ETRI Team focused on pure smartphone-based PDR and image-based localization, from the perspective of maintainability and sustainability.For Track 2 competition, PDR was applied as one single source of positioning.Since the competition venue of IPIN2018 was a large commercial mall, it was quite challenging for smartphone PDR to maintain the required level of accuracy during the course of the competition.To deal with this situation, an intrinsic PDR algorithm was tried to be enhanced.
PDR system consists of three basic steps: step detection, step length and heading estimation.To detect a step, peak detection using accelerometer magnitude is widely used in PDR.However, simple peak detection method only compares current and previous accelerometer values, which degrades detection accuracy.In order to eliminate unwanted steps, the detected timestamps between steps and threshold value are considered.Once a step is detected, then the relative distance between current and previous steps is estimated.One of the powerful factors to estimate step length is to use linear relationship between step frequency and step length; the faster a user walks, the longer the step length is [35].Another feature that is correlated with step length is the vertical acceleration.These two features are linearly combined for the stable step length estimation.Besides that, step length values are adjusted for certain walk patterns, such as corner walk, staircase walk, etc. Heading accuracy is closely related to the overall performance of PDR.Even though step detection and step length estimation are accurately done, the position accuracy is severely and gradually degraded when the device is not pointing along the walking direction.The direction of the device was then estimated using built-in gyroscope and accelerometer filtered in an attitude reference system (ARS).ARS exploits attitude calculated by gyroscope and accelerometer.Accelerometer output gives the roll and pitch angles under stationary condition.In order to combine the attitudes from gyroscope and accelerometer, Extended Kalman Filter (EKF) is applied to estimate the error states: attitude errors and gyro bias.Gyroscope bias is continuously updated and the acceleration vector is used as a measurement when stationary condition is detected.
In parallel, the use of semantic information was devised to assist heading drift, which causes a critical error in localization.Here, semantic information includes map information such as nearby angle values, floor change zone, etc. Dynamic heading adjustment was applied using the semantic information, and it was proven reasonable performance in the ETRI team site.
Looking back at the result, the performance of the approach was affected by the limited time for exploring the competition site and data verification.We found to main reasons of unsatisfactory performance at the time of competition.One is misaligned map information, and the other is delayed logging of estimates for waypoints.Although the error compensation using external information worked well, it was challenging to endure such a long path without any other positioning aid.

This technical description presents the indoor positioning technology that was implemented based on PDR and Map Matching for the Track 2 of IPIN 2018 competition by Kyushu team.
Since the chest has fewest unexpected movements in the upper body while walking compared with a hand or a head, it was selected as a novel IMU installation position.As well as other PDR frameworks, a step is first detected, and then a step length and direction are computed from the temporal sequence of accelerations and angular velocities.To adapt the PDR system to the chest-mounted IMU, novel regression model is proposed based o step length and direction estimation.In addition, an efficient map matching algorithm based on particle filter with barometric pressure was integrated.This largely suppresses the error accumulation on the position and heading of the pedestrian in the building when the step estimation causes some error.For the development, NGIMU 1 was used.It can be wirelessly connected to a laptop so that the computation was performed on the laptop.The detailed description has already been published [36], and the source code is available online 2 .

a: Step estimation
To separate the sequential data into the steps,an approach based on filtering and thresholding for the norm of the acceleration was ´´used.This process is the same as that in other PDR systems.The norm of the 3D acceleration in the sequence is first computed, and a low pass filter is applied to it.Finally, the peeks are extracted because they correspond to the moments when the foot hits the ground.In other words, one step is defined as the interval from a peek to the next one.
The next process is to compute the step length and direction.Fig. 11 illustrates the conceptual figure on the velocity changes of foot and chest while walking.For the footmounted IMU, zero velocity update is useful because the foot can be fixed while it is attached on the ground.However, the chest normally does not stop while walking, and its velocity change is periodic.From this characteristic, it was empirically formulated that the displacement of one step can be computed by: where a(t) is the 3D acceleration at time t, the range of the integral is the duration of one step, and K is a scalar parameter that needs to be calibrated for each user.In particular, K is always negative because the chest is first decelerated, and then accelerated in one step.Since the orientation of the IMU is updated by using the Madgwick method [37] all the time, the acceleration here is described in the world frame where z axis is parallel to the gravity direction.This means that the integral of the accelerations with K in x axis and y axis corresponds to the moving length and direction.With Eq. 6, it can simultaneously be estimated the step length and direction from the accelerations without estimating the velocity to avoid the drift error.
b: Map matching A particle filtering based map matching algorithm was used with a back tracking test [38].The following components for multi-floor PDR were selected to constraint the movement on the map: • line is used for the wall that can not be crossed.
• zone_no_particle represents the region where pedestrians cannot come into, and therefore the existing particles will be deleted, and new particles can not be generated.
• zone_update_altitude represents the region where the altitude can be changed such as stairs, escalators, and elevators so that the altitude will be updated according to the movement.• arrow is used for explicitly describing the direction of the possible movement at zone_update_altitude such that arrow_up connects to the upper floor, arrow_down connects to the lower floor, and arrow_both connects to both floors.A map editor was also implemented, whose input was the map images, and output was a set of files which contains the multi-floor building information.The users simply input points and connect them to create the map.

3) TUM Team
The high level architecture of TUM team system consists of seven components: 1) User's Context Extraction; 2) Map, which is responsible for mining the corresponding Open Street Maps Model; 3)Pedometer, that is responsible for detecting steps; 4) Initial Direction; 5) the Direction estimation; 6) Particle Filter, responsible for the fusion of the aforementioned components; and 7), the Visualize Location, that is responsible to visualize the estimated location (see Fig. 12).Map is responsible for parsing the map and for identifying elements that belong to walkable areas, such as stairs, corridors or pathways.
Grammars enhance map elements using established standards, like based on standards regarding the minimum width of corridors.Particle Generation generates a set of randomly positioned particles S k t = [x k j , w k j ], where x k j is the j th particle at time k and w k j is the normalized weight of the particle, with uniform prior distribution, as described by the Monte Carlo simulation.Particle Filter.The proposed filter is used to model the probability density of the state vector X k = [x k , y k , θ k ] T at step k by N p particles.According to [39]- [41] and assuming a first-order hidden Markov model, the posterior filtered density p x k |z 0:k is approximated by: where z 0:k defines the measurement vector for the time steps 0. . . ., k, δ(•) stands for the Dirac distribution, x k denotes particle state and w (j) k denotes the normalized weight.The conditional probability is P (S t |z t , S t−1 ) = P (z t |S t , S t−1 ) where S t and S t−1 are the current and previous state vectors respectively.z t is the observation, which in the proposed case is z t = min(S map − S t ), where S map is a vector of all points in the map.Motion Prediction is an implementation of the dead reckoning process where the motions are modeled by: where x k , y k and θ k are the coordinates and direction at time k.x k−1 and y k−1 are the previous coordinates.ρ is the traveled distance and θ k is the followed direction.
In the Update phase, each particle's weight is computed based on the assigned probabilities, a model that describes a set of restrictions and a motion predicted mechanism.Hence the weight of the i th particle at time k, w k is defined by: where p X describes the conditional probability of the i th particle being at location X at time k given the nearest distance δ to the nearest corridor.p describes the conditional probability of the particle being at location X, given its current heading direction θ.Finally, p X (i) k |α k is the conditional probability of the particle being at location X at time k given the current activity α.Resample.For proper resample, two additive Gaussian noise models are used on top of the motion prediction mechanism, in order to compensate direction and step length uncertainty.Initial Direction.In order to extract an initial direction, the phone's accelerometer and magnetometer are used to calculate the phone's pose and later the rotation matrix (roll, pitch and yaw angles).Direction Estimation For the direction estimation, a low pass filter is first applied to remove human motion noise and then the angular velocity is computed.Positive or negative sign to the values is assigned to indicate whether there is a right or left turn respectively.
Step Detection.In the step detection module a peak detection module is applied, which detects peaks caused by the action when the heel strikes the ground.Visualize Location.After calculating the new positions of the particles using the dead reckoning algorithm, weighted through a fusion with the map and the user context, the weighted average of the particles position is displayed as the final position to the user.Context Extraction.This module is responsible for collecting data, filtering noise, segmenting sensor values, extracting features and for the classification of these values in a series of activities.

4) UPJS Team
The UPJS team approach focuses on indoor positioning using low-cost smartphones with built-in sensors.Floor plans are assumed to be available and no additional building infrastructure is required.A relative position is determined using Pedestrian Dead Reckoning (PDR).Bayesian filtering is introduced to deal with the uncertainty from the noisy sensor measurements.A step is detected from acceleration data, which triggers a computation of a new position estimation.The step heading is obtained from filtered values of the device orientation information.The positioning is performed on a single floor.If a floor change is detected using barometer measurements, the system is reset.In this case, a new initial estimation is set to the predefined location from a set of all possible transition platforms (e.g., elevators, stairs) in the building and previous location estimates.
Our main research focus is on the Bayesian filtering especially on grid-based methods [42].Prediction and update are two stages of the filtering, where the prediction spread the probability density function according to the transition model and update stage decreases the uncertainty using measurements and observations.The grid-based filter approximates a continuous state space by a grid of regularly distributed isolated points.Values of conditional probability density function are computed only at these grid points.The method has high computational demands making it suitable for realtime indoor positioning only by assuming a low-dimensional system state.In this approach, every state is defined by a position in 2D space.The state transition is performed using the PDR with a fixed step length and the obtained step heading in addition with the noise model.The transition between two states represented by grid points is accomplished using a convolution with pre-computed convolution masks for various possible step lengths and directions.The update stage resets the belief value to zero for all grid points denoting inaccessible positions and movements, e.g., locations at obstacles, outside the map, motions through walls.
Walls and convex polygons composing accessible positions in the building are manually labeled in the floor plans.The map model is automatically generated and the floor plan is tessellated into a regular grid, where a point represents an area of 33 × 33 cm 2 denoting a single grid cell with the belief value.The grid-based filtering approach is elaborated and the positioning error introduced by discretization of the map is reduced using a centroidal grid approach.The same procedure is performed for the belief computation.However, the current position is not always represented by a center of a grid cell, but it is selected from the set of points in a finer grid according to the transition model.A single grid cell consists of 11 × 11 finer grid points.
The system proposed by UPJS team fuses PDR and map model using the grid-based Bayesian filtering.The analysis of the competition results show the applicability of such approach.However, even with the calibrated step length for the user, the main drawback of the solution is that the fixed step length causes the system to be less robust to different paces and walking styles in a realistic scenario.

C. TRACK 3: SMARTPHONE BASED POSITIONING CHALLENGE 1) ARARADS Team
AraraIPS is Arara's proprietary indoor positioning technology.Arara is engaged in developing advanced knowledge solutions and producing high-quality technology to address modern business and industry challenges.AraraIPS has been one of the central issues in Arara agenda for the last two years.It is a functional indoor positioning system on its way to become a commercial solution in the next months.
AraraDS team approach to indoor positioning has four distinctive characteristics: 1) it is based on a cartographic paradigm (fingerprinting); 2) it uses a discretization of the predicted floor/building, 3) it is measurement-agnostic (i.e. its abstract formulation is not specific to any kind of signal or measurement such as WiFi, magnetic field, BLE, etc.); and 4) it exploits measurement history.In the following paragraph it will be shown how AraraIPS works.
Fingerprinting has become a popular technique for indoor positioning.Traditional positioning methods are generally based on principles such as trilateration or triangulation, in which sufficient geometric information (with respect to reference landmarks) is used to compute the position of the object to be located.These methods are based on strong hypotheses that usually do not hold in the dynamic, cluttered context of indoor spaces, like the need of a clear line of sight between the object and the reference landmark to establish a functional one-to-one relation between the distance and the signal's intensity.
The system also relies on the discretization of the indoor space.In practice, this means that a graph is built from the map of a venue, in which nodes are possible locations where the tracked device can be found and edges connect neighboring nodes.This effectively turns the positioning problem into a classification one, in which the prediction is one of the finitely many possible locations.It has the further advantage of ruling out inaccessible locations (e.g.walls) and thus avoiding complicating the prediction task.
The final part of our system enriches the information available in the prediction module by including the measurements' history, exploiting the fact that measurements closely related in time will be strongly correlated.Thus, it is expected the prediction to narrow in on the true position with a greater confidence as time goes by.The mathematical formulation of this idea is the use of a random walk on the graph of the venue, in which transition probabilities at a given epoch depend on the measurements at the same epoch.This dependence is made precise by the underlying measurement and node transition probabilistic models, which in the proposed system authors choose not to disclose.

2) mCLIPS Team
Team 'mCLIPS' adopted an empirical approach, only based on WiFi, for indoor positioning.In this approach, some important features of the received signal strength indicator (RSSI) vector are selected to exploit the similarity measure and index order of the access points (APs) in the RSSI list of a WiFi device [43].The basic principle of the proposed VOLUME 4, 2016 FIGURE 13.Sample graph on the competition scenario, each blue dot represents a graph node approach is the following: instead of relying only on the RSS similarity measure, the system takes into account other features available in the RSSI list and their efficiency weight to identify candidate positions with high precision and robustness.The selected features have several dimensions.The K value and the kRMSE parameter calculation are selected to control the diversity effects of WiFi environments [44].To exploit the spatial distribution of candidate reference points around the ground truth, another parameter called centroid distance is used [45].To control the effect of the diversity of different mobile devices, the entropy parameter is adopted from ranked based approach [46].In addition, these features are combined to compensate complementary weaknesses.As a result, the algorithm of the proposed fingerprint matching approach is more efficient and robust than algorithms already available in the literature for fingerprinting purposes.To find the best match among multiple candidate fingerprint locations with a minimum error distance, the proposed algorithm along with conventional RSSI similarity calculation in fingerprint matching process also calculates the order difference of the APs in RSSI list with respect to the signal strength at a particular reference point (RP).The following sections describe the steps of the proposed technique.
• The first step is the selection of K. K selection applies to the WiFi infrastructure present in the environment and is directly proportional to the good APs in the environment.According to the Good AP definition in [47], APs with consistency in presence at particular reference points are of great importance compared to others that disappear in consecutive scans.Eq. 10 and Eq.11 describe the selection of K.
G RPi and T RPi are Good AP count and Total AP count to each cell of the fingerprinting DB, respectively.K RSSList is the total AP count in a particular RSSV.The variable C is the standard deviation.
• From an empirical point of view, it has been observed that the existence of RP corresponding to the ground truth is high among the RP candidates having a high number of AP matches.Therefore, the maximum match count (MMC) of APs relative to the number of neighbors is an important measure used to avoid outliers that have a high correlation but a lower value for the number of neighbors.• The root means square error (RMSE) is one of the important parameters that plays a key role in the similarity calculation of fingerprints.Experimentation shows that in indoor radio environments, some areas lead to the selection of outliers with low RMSE values due to a strong correlation between some access points with low RSSI values.Therefore, to prioritize the candidate locations having a higher number of AP mappings, a kRMSE of candidate location is calculated that includes an additional division with the number of APs compared (i.e.MMC) to RMSE calculation.• Another important parameter for filtering the estimated positions is the distance from the center of gravity [48].After computing kRMSE of each candidate, the average location of the top ordered N RPs relative to the minimum kRMSE value is selected as centroid point.In the proposed algorithm, the position of the centroid is calculated by taking an average of the coordinate values (X, Y ) of each selected candidate position.The Euclidean distance of a candidate RP from the calculated centroid is called the centroid distance.This distance from the centroid is used to define the RP priorities that are closer to the centroid point.• Next, the entropy error is another important measure after kRMSE to estimate the similarities of each candidate location RSSI list in the database against the RSSI list received online.The proposed algorithm calculates ∆ V ji for each corresponding AP in both RSSI lists, first by sorting the two RSSI lists relative to the RSSI level, then measuring the difference of AP index in both lists (Eq.12).In addition, the order difference of an AP with a stronger RSSI is minimized by multiplying it by the RSSI level (RSSI V ji ) divided by -100.Empirical results show that the position accuracy performance of the proposed entropy parameter outperforms the ranked based approach [46].Therefore, entropy helps in lim- iting the strong candidates around the ground truth by calculating the order difference of each AP with respect to its signal strength.
where k ≤ K where I V ji and I Cji are indices of matched APs in online and candidate RSSI list from DB, respectively.• Finally, the FACT calculation is performed to select the final candidate location.FACT calculation begins with the normalization of the parameter values of each selected candidate.Then the FACT is calculated as a product of kRM SE with the sum of Entropy and D cntr , as detailed in Eq. 14.The candidate RP with the least value of FACT is selected as the final position estimated by the proposed algorithm.
3) EGEC Team The algorithm used by EGEC Team is based on PDR and map matching.WiFi RSSI information and magnetic field strength information are also used to build the fingerprint databases, to provide the initial position and improve the algorithm's precision and stability.The sensor information used in this algorithm includes the inertial measurements (accelerometer and gyroscope), WiFi RSSI, magnetic field strength and air pressure.In particular, the map information provided by the competition organizers is fully utilized in the map matching algorithm to correct the pedestrian's position and direction.Figure 14 shows the technical route of the algorithm.

I. THE INDOOR LOCALIZATION ALGORITHM BASED ON PDR AND MAP MATCHING
This section describes the technical route of the algorithm used in the competition of Track 3.This algorithm is based on pedestrian dead reckoning (PDR) and Map matching.Meanwhile, Wi-Fi RSSI information and magnetic field strength information are also used for building the fingerprint databases, to provide the initial position and improve the algorithm precision and stability.The sensor information used in this algorithm include the inertial measurements (accelerometer and gyroscope), Wi-Fi RSSI, magnetic field strength and air pressure.In particular, the map information provided by the competition is fully utilized in the map matching algorithm to correct the pedestrian's position and direction.

A. Pedestrian Dead Reckoning (PDR)
The PDR algorithm mainly uses mobile phone sensor measurements like accelerometers, gyroscopes and magnetic to estimate pedestrian step number, step length and direction, and then calculate the pedestrian walking trajectory and position.The initial position used in the PDR algorithm is given by Wi-Fi RSSI fingerprint.Meanwhile, the air pressure is also used for the floor switching.
The PDR algorithm contains step detection, step length estimation and direction estimation algorithms.Compared to traditional PDR algorithms, the algorithm uses an optimized physiological model of pedestrian gait, which is more adaptable and stable for pedestrian step detection and step length estimation.

B. Map Matching
Because the PDR algorithm result contains cumulative errors, which may locate the pedestrian position outside of the indoor road network and cause greater calculation errors.
In that case, after calculating the pedestrian position, a map matching algorithm is used to match the current pedestrian locations with the indoor GIS maps provided by the competition.
The map matching algorithm used in this manuscript is the optimized ST-matching algorithm, which comprehensive use the indoor road network location, direction and pedestrian speed.The optimized ST-matching algorithm can fully utilize the map information provided by the competition and is more suitable for the indoor positioning and navigation.

C. Fingerprint Matching Algorithm
In addition, the algorithm uses Wi-Fi RSSI, magnetic field strength to build Wi-Fi fingerprint and magnetic fingerprint.The Wi-Fi fingerprint and magnetic fingerprint are used to provide initial position and correct the pedestrian directions and position errors.In this manuscript, the WIFI fingerprint matching algorithm is the cosine similarity algorithm, and the geomagnetic fingerprint matching algorithm is the contour matching algorithm.The fingerprint matching algorithms can significantly improve the accuracy and reliability of the algorithm.

D. The fusion filter
In the algorithm, a Kalman filter is designed to combine different results of these algorithms and provide real-time, accurate and stable indoor positioning results.The PDR algorithm (Fig. 15) mainly uses mobile phone sensors like accelerometers, gyroscopes and magnetometers to estimate pedestrian's step number, step length and direction, and then calculates the pedestrian walking trajectory and position.The initial position used in the PDR algorithm is given by WiFi RSSI fingerprint.The air pressure is also used for the floor switching.
The PDR algorithm contains a step detection, a step length estimation and a direction estimation algorithms.Compared to traditional PDR algorithms, it uses an optimized physiological model of pedestrian gait, which is more adaptable and stable for pedestrian step detection and step length estimation.

VOLUME XX, 2017
given by Wi-Fi RSSI fingerprint.Meanwhile, the air pressure is also used for the floor switching.The PDR algorithm contains step detection, step length estimation and direction estimation algorithms.Compared to traditional PDR algorithms, the algorithm uses an optimized physiological model of pedestrian gait, which is more adaptable and stable for pedestrian step detection and step length estimation.b: Map Matching PDR algorithm is affected by cumulative errors, which may result in locating the pedestrian outside of the indoor road network with large calculation errors.In that case, a map matching algorithm is used to match the current pedestrian location with the indoor GIS maps provided by the competition's organizers.
The map matching algorithm used in this manuscript is the optimized ST-matching algorithm, that fuses the indoor road network location, direction and pedestrian speed.It fully utilizes the map information and is more suitable for the indoor positioning and navigation.

c: Fingerprint Matching Algorithm
The algorithm uses also WiFi RSSI, magnetic field strength to build WiFi and magnetic fingerprints.They are used to provide the initial position and correct the pedestrian's direction and position estimation errors.The WiFi fingerprint matching algorithm is the cosine similarity algorithm.The geomagnetic fingerprint matching algorithm is the contour matching algorithm.These matching algorithms can significantly improve the accuracy and reliability of the solution.

d: The fusion filter
A Kalman filter combines the different algorithms' estimates and provides real-time, accurate and stable indoor positioning results, as shown in Fig. 16.

4) HFTS Team
The HFTS algorithm used in the 2018 competition is based on WiFi and pedestrian dead reckoning (PDR), and a particle filter (PF) (Fig. 17).It is an evolution of the algorithm deployed at the 2016 competition [49] and the 2017 Competition [50].
For the dead reckoning, heading estimation is performed using compass and gyroscope data.The gyroscope is able to

D. The fusion filter
In the algorithm, a Kalman filter is designed to combine different results of these algorithms and provide real-time, accurate and stable indoor positioning results.detect heading changes quite accurately on a short timescale, but will drift in the long-term.In contrast, compass heading is subject to strong local magnetic perturbations but shows no drift on a long-term scale.Therefore, gyroscope data is used to detect immediate heading changes.The heading relaxes to the compass direction with a certain time constant of some seconds.
Step detection is performed by peak detection of the accelerometer data.
WiFi positioning is performed using the scalar product correlation fingerprinting algorithm [51] which is based on k − nn and the cosine similarity.Prior to the estimation of the unknown competition track, a radio map is obtained by evaluating the reference data provided by the competition organizers.These data are enriched with true ground positions and RSS readings.Therefore obtained RSS vectors can be related to a position.As the true ground readings occur only rarely, the actual reference position between two reference points is interpolated by PDR.
The actual positioning process comprises 2 phases.In the first phase RSS positioning is performed with the radio map.RSS readings are obtained at a rate of one reading every 4 seconds.For each reading a position is estimated.The floor is also estimated by RSS.If the actor is moving, the position will change considerably within the 4 seconds between two RSS readings.Therefore, the position between the two readings is interpolated using PDR.The heading offset and the actual step length are not known at this stage.However, since the RSS position estimates exist for the startand the endpoint of the 4 second period, the step length and heading offset are adjusted such that the track connects the two points.
In the second phase, positioning is performed employing PDR and a particle filter.
Step length estimation and heading error-and drift compensation are performed by the particle filter using the information of floor plans to detect the most likely path (see also [52], [53]).The particle filter contains a constant number of particles.Besides the position, a particle state also comprises individual step length and heading offset values.The filter is updated each time a step is detected.All particles are moved according to the estimated heading, individually modified by the particle specific offsets.Resampling is performed as follows: particles who collide with, for example, a wall, are replaced by new ones.For collision detection, the provided floor plans are used.New particles are seeded at the position of an existing particle, but with randomized step length and heading offset value.The global heading and step length are recalculated by averaging the heading values and step length of all particles.The reported PDR position is the averaged position of all particles.
Depending on the building layout (e.g.open spaces, corridors, room sizes, etc.), the PF results alone may not be sufficient for absolute position determination.Therefore, the step length and heading are also adjusted by the obtained WiFi position from phase 1: After each step the longitudinaland the lateral displacement between WiFi and PF position are evaluated and a configurable fraction of the displacement is added to the movement vector.
It is planned to further improve the algorithm by also considering magnetic fingerprinting.

5) TUM Team
The system proposed by TUM team, Ariadn, can be divided into two main modules.The first module is responsible for the localization and its functionality is embedded in a smartphone; and the second module is responsible for detecting and localizing landmarks and keeping up-to-date a classifier that enables "on the fly" localization.

a: The Localization Module
The localization module is responsible for localizing the user following a modified particle filter approach.The main components of this module, presented in Fig. 18, are now OSM Model.Existing information from Open Street Maps is imported to enable the generation of particles.Grammars.They are heuristics following standards that help to create a finer geometry.They are necessary due to the OSM limited information and they are inspired by [54].Particle Generation.In this module, the newly constructed map is restructured as a set of randomly generated particles, with uniform prior distribution, which follows the Monte Carlo simulation description.Initial Direction.The initial direction is estimated following an opportunistic approach, based on the users context (i.e."user transit indoors", "user is ascending stairs", etc.).When this is not possible, the direction is extracted from the compass after the phone pose (i.e. on hand, in pocket etc.) has been extracted.
Step Counter.In the step counter module, the current number of steps is estimated based on the repetitive pattern caused in the accelerometer from the human bipedal movement.
Localization.Ariadne provides horizontal and vertical localization, where the horizontal localization is based on Dead Reckoning, while the vertical is based on the barometric formula.Quantify Confidence.In every unit of time new particles are generated, weighted based on the probability of being the target's true location.Their weights are quantified based on map restrictions or model predictions.Visualize Location.Finally, the particle with the highest probability is visualized as the real location.

b: The Landmark Selection Module
The Landmark Selection module (Fig. 19) is responsible for extracting landmarks from raw sensor data that have been extracted while the user was localized from the localization module and for localizing the user once a model is available.Its components are: Training.Once there are enough data and with a regular interval, a classifier is trained on them and more precise localization is enabled.

6) UGENT Team
The core of UGENT team location tracking system is a route mapping filter [55] that is based on a motion model and the Viterbi principle, a technique related to Hidden Markov Models and backward belief propagation.The physical layout of a building is used to construct the most likely path instead of a sequence of independent, instantaneous estimates.This postprocessing filter ensures physically realistic trajectories and has been shown by independent researchers to outperform traditional approaches such as smoothing particle filters [56].During the initialization, the route mapping filter starts with N paths (e.g., 1000 starting points) that are located in a circle around the best first estimation given the first WiFi measurements and a fingerprint map based on the training data [55].These N paths are updated each time new WiFi measurements become available.The next candidate positions, starting from the current last points of the paths in memory, are determined based on a maximum walking speed (derived from a step counting algorithm using the available accelerometer data), and the walls and obstacles of the building (people cannot move through walls).Each path consists of a chain of grid points and a cost that indicates the probability of this path at this time step.The path with the lowest cost after processing all sensor data is the most likely trajectory.The cost of a path is the sum of costs based on various sensor measurements: WiFi RSS measurements, barometer, accelerometer and gyroscope data: WiFi.The RSS values are compared to the corresponding reference values in a fingerprint map and are weighed based on the estimated distance to the WiFi access points.The fingerprint map itself is based on all training data, grouped per BSSID and per grid point.Missing grid points in the fingerprint map are filled in based on interpolation of neighboring grid points (that do have measurements), with a certain upper limit, e.g., 5m.Barometer.The differences between the average pressure over a short (5 seconds) and a long window (30 seconds) are used to detect up or down floor changes when the difference exceeds a certain threshold based on the training data.Furthermore, a cooling-off period is implemented to avoid changing floors multiple times within a short time span.Paths that do no change their current floor in the right direction (by taking the escalator or elevator) get an additional cost assigned.By default, this additional cost is set to the floor height or floor penalty (15 m).Accelerometer, gyroscope, and mobile phone 3D orienta-These data are fused together and used to detect the step count, stride length and orientation between two locations updates; combination of these three gives the traveled distance and direction.The difference between this traveled distance and direction, and the distance and direction between a grid point and its parent grid point (i.e., the previous grid point of the path) are used as an additional cost.This penalizes paths that move when the accelerometer and gyroscope estimate that the user is standing still; and also paths that remain static when the user is moving; or paths that take a turn in the opposite direction.
The physical layout of the building is modeled by a shapefile, constructed with QGIS, the PDF maps with georeferenced points, and a plugin to reference the raster to the projected coordinate system by creating a GeoTIFF file.Each floor is represented by a polygon with the floor id as a feature value and likewise, the escalator and elevator are represented by a polygon with the start and end floor as a feature value (Fig. 20(b)).The route mapping filter uses this shapefile as input to construct a grid database with a certain grid size, e.g., 1 m, and a reachable database, consisting of the reachable grid points, for each grid point (Fig. 20(a)).The reachable grid points are the neighboring grid points that do not cross walls or obstacles, and the above or below grid points if they are located within the polygon of an escalator or elevator (and correct floor level).This grid and reachable database are used to select the next candidate positions in the route mapping filter, given the current last points of the paths in memory and the time that has passed since the last location update.Floor plan modeled as polygon with details of the grid.

7) WHU Team
The method used by Wuhan University team utilizes data of many sensors available in a smartphone.The main method combines PDR, Wi-Fi fingerprinting and magnetic matching.
The PDR algorithm processes accelerometer, compass, gyroscope and barometer data.Similarly to PDR algorithm, compass and gyroscope signals are used to estimate heading and accelerometer signal is used for step detection.In contrast to traditional PDR, barometer data is used as an important source for the altitude estimation to detect much more precisely the floor.
Since the competition organizers have provided a training set, the team took full advantage of the WiFi signals to construct a fingerprinting database.WiFi fingerprinting method is an attractive indoor positioning method but the training data was limited.That is the reason why magnetic information was fully used.It is important to note that the geomagnetic sequence and fluctuations are often similar people pass through the same path.These anomalies are, therefore, highly reliable.The position of the reference was used to control the position drift error of the inertial navigation system.

8) YAI Team
YAI team fingerprinting method first leverages the three-axis accelerometer to build trusted known points associated with APP time.Next, the team uses WiFi received signal to reconstruct the WiFi fingerprints.Finally, the system compares both, the Euclidean distance and the similarity of the received WiFi behavior metrics to compute the final localization results.

a: Step detection
The training dataset gives the coordinates (longitude and latitude) of some known points, i.e.POSI, but these POSI points are sparse in the space domain.To build a WiFi fingerprint, coordinates of many other points than the known POSI points have to be estimated.Therefore, first, the system assumes that the walking path between two consecutive POSI points follows a straight line.Then, the sensed data of the three-axis accelerometer is used to estimate the number of steps between two known POSI positions.This can be done by finding the local maximum points of the magnitude of the output of a three-axis accelerometer between two POSI points.The magnitude can be expressed as: where x i , y i , and z i are the values of a three-axis accelerometer at epoch i.The symbol E {•} denotes the averaged operation.Lastly, we use inter-and intrapolation to estimate the locations of the interpolated points between two POSI points.At this stage, we have a table which lists 4-tuple for each record: (APP Time, Longitude, Latitude, Floor Number).

b: Generation of the fingerprinting table
Next, the received RSS values from each access point (AP) are associated with the 4-tuple table.Note that the APP time of receving the RSS values is not synchronized to the 4-tuple table.However, over short time duration, the location of a walking man should be pretty similar.Thus, the location with the closest APP time is chosen to generate the fingerprinting table.

c: Positioning Method
The positioning method simply consists in comparing a Ntuple received RSS values with the WiFi fingerprinting table.In addition, a metric called "Similarity of the WiFi receiving behavior (S value)" is defined.It is used to measure the distance of the N -tuple received RSS values with the records in the WiFi fingerprinting table.Since the received RSS values with each AP are not stable, it is better not to compare all the entries of the fingerprinting table with the N -tuple received RSS values.Instead, the entries of the fingerprinting table, which have the highest S value, are compared with the N -tuple received RSS values.The two proposed "similarity of the WiFi receiving behavior metrics", S 1 and S 2 , are: where x j denotes the j-th entry of the received N-tuple; y ij denotes the j-th entry of the i-th record in the fingerprinting table.sgn(•) is a sign function defined as follows: and I(•, •) is an indication function defined as follows: Finally, the Euclidean distance is calculated between the Ntuple and the records in the fingerprinting table which has the largest first two S values and the location of the N -tuple with the smallest Euclidean distance is obtained.It is important to note that, in general, S 1 is used as the similarity of the WiFi receiving behavior metric.However, when the sparsity of the N -tuple is high, S 2 is adopted as the similarity of the WiFi receiving behavior metric instead.

1) AOE Team
The system proposed by AOE team is a human foot-mounted PDR.Fig. 21 gives an overview of the solution comprising three different modules.
The bottom module is an IMU device with four sensors: an accelerometer for measuring acceleration a b t+1 , a gyroscope for measuring angular rate ω b t+1 , a magnetometer for measuring magnetic field of the Earth o b t+1 , and a barometer for measuring barometric pressure of the Earth, with which the person's altitude, h t+1 , can be calculated.
The middle module includes five components: the Stance & Still Phase Detection, the HDR, the ZUPT, the ZARU, and the Earth Magnetic Yaw.Meanwhile, the Stance & Still Phase Detection includes two components.The GLRT detector algorithm is used under slow and normal pedestrian gait speed condition and the HMM detector algorithm is used under high dynamic and fast pedestrian gait speed condition, such as fast walking.When the Stance & Still Phase Detection detects the stance and swing phases of human foot gait from IMU's data, the HDR, the ZUPT, the ZARU, and the Earth Magnetic Yaw utilize these data to estimate the error vector m t+1 (δψ t+1 , δω b t+1 , δv t+1 ) (Fig. 21), where δψ t+1 , δω b t+1 and δv t+1 denote the bias error of yaw, the bias error of angular rate and the bias error of velocity at time t + 1 , respectively.m t+1 (δψ t+1 , δω b t+1 , δv t+1 ) is the most important input data for successfully bounding errors in the EKF algorithm.Since the EKF mechanization part processes dynamic data that are related to position, velocity, and attitude, it relies on the Stance & Still Phase Detection to apply the HDR, the ZUPT, the ZARU and the Earth Magnetic Yaw updates.
The top module includes an INS and an EKF.The INS system alone can't cope with the IMU drift.The EKF, along with a properly constructed sensor fusion scheme, can however estimate the IMU biases.Therefore, it can help the INS in reducing the IMU drift.To mitigate possible gross errors in the measurements, a fault detection method based on the innovation vector is adopted.It can effectively detect and remove the gross errors to ensure that only accurate measurements are fused.

a: Experimental evaluation
In this section, the proposed algorithm is implemented in the indoor experiment, in order to evaluate the accuracy of the position estimation of the foot-mounted PDR system.One of the experimental data result is shown in Fig. 22.The errors in the horizontal plane for the three groups of test data are detailed in Table 2.  AOE algorithm has been tested in the real indoor environment.The experiment shows that the algorithm can effec-tively eliminate the error accumulation of the IMU device, with a high accuracy.

2) KIT Team
In Fig. 23 an overview KIT team tightly-coupled INS/GNSS sensor data fusion approach is illustrated.The current position, velocity and attitude are propagated with a strapdown-algorithm considering the specific force and gyroscope measurements.The measured inertial signals are corrected with estimated sensor bias values.Zero-Velocity-Updates (ZUPTs) are generated in the detected midstance phases of the foot by a finite state machine based gait phase classifier [57]- [59].In contrast, horizontal ZUPTs and delta ZUPTs are applied during stance phases using an elevator.Absolute velocity measurements are generated during stance phases on an escalator [60].If GNSS pseudo-range and Doppler measurements are available, the kind of aiding depends on the current classified motion state.To enable accurate position estimation in urban scenarios, an integrity check of the received GNSS data is essential [61].Based on the described measurements, corrections of the navigation solution and the bias values of the inertial sensors are calculated by the Error-State-Stochastic-Kloning-Kalman-Filter.

a: Gait Phase Classifier
The proposed finite state machine based motion classifier method is founded on biomechanical knowledge and medical research findings of the human gait.Because the foot module is solely mounted on one foot, it is appropriate to model four basic motion states for forward motion, as shown in Fig. 24 and described in detail in [58].The transitions and states representing backwards motion and the detection of walking and running are explained in [59].b: Elevator and Escalator Detector KIT team approach is able to detect and separate elevator from escalator movements [60].In addition, the elevator and escalator rides are subdivided in different sub-states like acceleration phase, constant velocity phase and braking phase.The elevator and escalator detector is realized with two separate finite state machines; one for the recognition of elevators and the second for the detection of escalators.The finite state machines have different states and transitions to switch between the states.

c: GNSS Augmentation
A tightly-coupled INS/GNSS integration is used for the pedestrian navigation system.The precise relative positioning INS is fused with GNSS pseudo-range and Doppler measurements for absolute position and heading estimation.Unique is the decision depending on the classified motion state if Doppler measurements improve the accuracy of the navigation solution.In addition, the integrity of the GNSS signals are checked.The good knowledge of the system precise relative positioning inertial system is used to detect and exclude GNSS measurements with high multipath errors.This is essential for robust pedestrian localization in dense urban environments with outdoor-indoor transitions.

3) WHU Team
In the last several years, WHU team has been working on developing an accurate and reliable pedestrian navigation system (PNS), which will significantly increase the safety of military personnel and first responders.
The data sources used in WHU solution are IMU data and GNSS raw data.Magnetometer is not included due to its vulnerability to disturbance (especially indoors).The basic navigation algorithm adopted is quite simple and commonly consists of INS mechanization, zero-velocity detector, and a complementary Kalman filter (KF), as shown in Fig. 25.
The Foot-Mounted INS signals are integrated with GNSS signals in a loosely-coupled manner.To get rid of gross error as much as possible, satellites with small elevation should be discarded.Some measurements with poor quality based on the magnitude and covariance of the innovation vector can also be rejected.Even though the heading is not observable when a GNSS signal is not available, heuristic methods like zero integrated heading rate are adopted because it is not reliable and accurate enough for a statistical grade or better IMU.
The height precision should be sufficient to distinguish between different floors.Some constraints, employed effectively, mitigate the error drift along the vertical channel in the absence of barometer.When walking up and down stairs, the slope can be seen as constant in most cases.So the system is able to define a slope from every footstep, as shown in Fig. 26.In this way, the slopes that are already in the buffer can be used to constraint the current slope, as well as updating the slope buffer with this newest slope.
Another extreme case is the escalator.Normally escalator moves with constant speed, which means that nearly all the acceleration input comes from local gravity when users stay relatively static on the escalator.The gravity information can be fused in a tightly coupled manner, the measurement equation can be defined in error form as: By using this gravity information, the drifting error part can be controlled even when moving on the escalator.The only thing, that needs to be done, is to compare the magnitude of the specific force with the one of the local gravity before proceeding with the update to determine the availability of the measurement.Thanks to this approach, the tricky issue of escalator recognition can be circumvented and still obtain a good position's estimate on escalator.

4) YAI Team
The system proposed by YAI team is shown in Fig. 27.The positioning algorithm includes three steps: step detection, stride and walking-direction estimation.The first two steps utilize accelerometers and the last two steps exploit the data from the component of the accelerometers and the magnetometers.In addition, outlier samples observed in the accelerometers are ensure smooth and stable walking-step estimation.Multivariable linear regression techniques were used to estimate the stride for each step.The detailed operation of the proposed moving trajectory estimation algorithm is now described a: Acceleration Decomposition By averaging the accelerometer values, an averaged g-vector can be obtained over the past few seconds, which can be expressed by: where g 0 represents a unit vector along the vertical direction.g i is the i-th reading of the g-sensor; N is the number of readings in the window.This can be generalized by the concept of low pass filtering.The vertical component of the g-vector can be formulated as follows: Also, the vertical component of g i can be expressed by The horizontal component of the g-vector becomes Note that g = i is distributed in a 2D space.

b: Stride Module
Next, we calculate the step's distance using multi-variable linear regression equation for each step with: where ∆A = ∆A 1 + ∆A 2 ; ∆A 1 and ∆A 2 are the difference between the left peak-valley pair and right peak-valley pair, respectively; C 0 , C 1 and C 2 are parameter; X 1 is the first component of DFT of g ⊥ i in the period.

c: Walking Direction Calculation
Finally, the orientation of the STIM300 and HMC5983 are used to calculate the horizontal component of m-vector, which can be expressed as follows: where m i represents the i-th reading of the m-sensor.By calculating the horizontal component of e z = (0, 0, 1), we obtain: The angle between the smartphone orientation and the north is obtained by combining m = i and e = z as follows: If m = i × e = y • g 0 ≤ 0, walking direction is toward east, otherwise, toward west.The holding angle is figured out by using horizontal acceleration vectors.The horizontal acceleration is composed of forward vector, backward vector and lateral vector.Walking causes great acceleration in forward and backward directions, and small acceleration in lateral.By accumulating the horizontal acceleration vectors, the lateral vectors can be compensated and remains only the forward and backward vectors.

V. COMPETITION RESULTS
This section presents the practical course of the competition for the four tracks and an analysis of results achieved by the teams who participated in the competition.

A. PRACTICAL COURSE OF THE COMPETITION
For the on-site tracks, the teams were allowed to survey the evaluation area during the calibration day (Friday, September 21 st , 2018) in the regular opening hours of the shopping mall with the restriction of not disturbing the daily life of shopkeepers and customers.Accessing the shops was explicitly forbidden since the evaluation only covered the common areas, like corridors or car park area.The competition organizers were present at three different times on that day to provide important information like the initial waypoint (lat, long, heading) for each on-site Track (Tracks 1 and 2), act as the main interface with Atlantis shopping mall and to answer those questions from the competitors, if possible.It is important to highlight that the organizers did not disturb or help in the calibration of the indoor positioning solutions.
The evaluation of the Track 1 teams was performed on Saturday, September 22 nd in the morning, whereas the evaluation of the Track 2 teams was performed the same day, in the afternoon after lunch.All teams were requested to be present in a briefing meeting to inform them about the evaluation features and randomly assign the evaluation slot for each team.Each team had to be in the meeting room five minutes before the assigned time slot.Two people from the organizing team, the evaluators, accompanied the competitors to the initial point and ensured they arrived on-time for evaluation.Moreover, there were more people from the organizing team spread in the shopping center to ensure that the elevator, stairs and sitting places were available for all the competitors.The two main evaluators were in close contact with the teams during the evaluation.One of the evaluators acted as path discoverer, and she/he was a little ahead of the competitors during the evaluation.The main objective was to open the path in the presence of crowds in the shopping mall and visually indicate the next evaluation waypoint.The other evaluator remained close to the competitor.She/he gave indications about the next evaluation waypoint and checked that the competitor was acting according to the rules.Before starting, the competitors had a few seconds to initialize their system.Once the end of the evaluation path was reached, the competitor had to go directly to the meeting room with the two organizers and provide the position estimates.After the first round of evaluations, the teams were informed that they had a second chance to evaluate their solution.If a team decided to run the course a second time, only this second course would be evaluated.
For the off-site tracks, Track 3 and Track 4, competitors had to provide the position estimates the latest on September 14 th , several days before the conference and competition.The evaluation trajectory was recorded by the track chairs in advance following a similar approach than the one reported for the on-site tracks.i.e., the competition organizers sur-veyed the area and collected the required data for the system calibration under realistic conditions (similar than the ones for the on-site tracks) and, after that, they recorded the data for positioning algorithms on the evaluation path simulating a real-time on-site evaluation.
The same format was expected for all the competition tracks.For each trial, the teams had to submit a CSV file, where each row represented a single estimated position.Each estimation had to include the timestamp in ms, the WGS84 longitude and latitude in decimal degrees with at least 9 decimal digit resolution, the floor number as an integer value (0 : Ground Floor, -1, 1, 2), the waypoint index in integer (waypoint number from 1 to N. 0 represents no landmark.Each specific integer represents the specific waypoint).The comma (",") was used as data delimiter and headers were not allowed in the CSV file.In Track 3, the timestamps for the waypoints were not provided to the competitors to avoid giving them many clues about the evaluation strategy.In that track, the competitors had to provide position estimates at a 2Hz frequency, synchronized with the beginning of the evaluation trajectory according to the timestamp provided in the evaluation log file.

B. ACCURACY SCORE ESTIMATION
The "accuracy score" was computed for each team by comparing the estimated coordinates with the reference coordinates of the waypoints marked on the ground and they were marching on during the competition.This metric combines the floor detection accuracy and the horizontal positioning error. where • P R are the ground truth horizontal coordinates • P E are the horizontal coordinates estimated by the competitors • P R − P E is the horizontal error and it is computed as the Euclidean distance between the ground truth and the estimated position provided by the competitor in the 2D space.
• p is used to penalize errors in estimating the floor and is set to 15 m.
number and the predicted one.The positioning error, ε, is computed for all way-points marked on the ground that define the path of a specific challenge.The "accuracy score" s is given by the third quartile error, in meters, of a cumulative error distribution function for all ε by: s = 3 rd quartile {ε} The team with the lowest score won the challenge.A total of forty-nine world-wide teams registered for the competition.Although thirty-four teams finally participated in the competition, only the twenty-seven shown in the table agreed to publish the results.Some companies are not allowed to report the results and a few teams could not complete the evaluation path.Fifteen teams withdrew their proposal and did not participate in the competition.

C. ANALYSIS OF THE ON-SITE CHALLENGE RESULTS
Most of the teams that competed for the on-site challenges performed a comprehensive pre-survey of the shopping mall the day before the competition.The footpaths were kept secret until the competition day, therefore the teams surveyed rather large areas in a short time.
In Track 1, the accuracy score ranges from 11.7 to 19.9 meters, whereas it ranges from 5.5 to 37.5 meters for Track 2. Surprisingly, some self-contained positioning solutions, not using a camera, performed better than the ones using video processing.Specular reflection was not affecting the performances with a fully indoor path and not too many obstacles in the camera field were present since Track 1 competition happened in the morning with a shopping center not too crowded.
The most popular technologies for Track 1 were visible light, augmented and mixed reality (ARCore, ARKit), inertial sensors, segmented maps and images.For Track 2, they were mainly restricted to inertial sensors, barometers and indoor maps.Popular processing methods for Track 1 are based on particle filter (segmented maps and images) and hybridization (PDR and optical flow).The same processing methods were adopted for Track 2 but complemented by highly accurate MEMS error calibration and machine learning for step length estimation and activity recognition.Competitors carried their positioning systems, providing a more favorable context for the success of machine learning methods.

D. ANALYSIS OF THE OFF-SITE CHALLENGE RESULTS
In contrast to the on-site competitors, the teams that participated in the off-site competition had around 16 weeks to calibrate their solutions with the data provided by competition organizers.It is worth to mention that the data format and strategy to collect data was new to most of the competing teams, that had to adapt their existing solutions to the provided data.
In Track 3, the accuracy ranges from 0.90 to 12.30 meters whereas it ranges from 1.30 to 90.80 meters for Track 4.Although the accuracy for the Track 3 (based on smartphone data) was generally better than for the Track 4 (based on PDR), the competitors of Track 3 had a huge advantage in form of many geo-referenced training data inside the shopping mall to, for instance, generate the WiFi radio maps.In contrast, the Track 4 competitors did not have such kind of training and validation data.
The most popular technology for Track 3 was WiFi fingerprinting, which has been the core technology of most proposals.The competitors have improved the fingerprinting approach by refining the position estimates with the data provided by the other available technologies, this additional information mostly came from inertial sensors and the magnetometer.In fact, the most accurate systems combined data from multiple sources, including magnetometer.
For Track 4, only IMU, magnetometer records and GNSS signals were used.The competitors processed the several hours MEMS data-logs collected in static conditions to identify and calibrate the sensors' noises.The most popular mechanization was the "strap-down" one completed by complementary filters and stochastic cloning Kalman filter.To mitigate the impact of sensors drift in inertial signals, competitors applied machine learning techniques based on mobility patterns and generalized likelihood ratio tests to identify specific walking gait periods and correct the positioning error.Remarkably, precise results have been obtained for conveyer belt and stairway passages.

E. LESSONS LEARNED
The most important lesson learned is that moving the competition site to a shopping mall has attracted many teams from academy and industry to participate in the proposed challenges.A total of 15 teams participated in the on-site tracks, with only one not able to provide the position estimates.The number of submissions for the on-site tracks was a success compared to previous editions were only 3 teams (2015) [62], 11 teams (2016) [63], [64] and teams (2017) [11], [65] successfully provided the estimates on the evaluation path.Providing an evaluation schedule with tight slots was an additional motivation for competitors to ensure they had a robust solution.Moreover, 19 teams participated in the off-site tracks.The participation in the off-site tracks has also been notably higher than in previous editions, which demonstrates that selecting a shopping mall was a good choice.
Organizing a two-day competition is not trivial and it involves some additional workload for the conference organizers.The most important steps are detailed below: • Agreeing to a set of rules for the competition with the venue authorities and adhering to them.A detailed agreement with the competition venue, the shopping mall, in this case, is required to organize the competition.This agreement must contain all the competition features and requirements should be detailed and had to be formalized many months beforehand.Thus, every aspect has to be well-documented and agreed with the shopping mall managers with little margin for any unexpected events or special requirements.Hopefully, some key information about the on-site and off-site competitions was collected more than one year before the competition and the most critical aspects were clarified by the organizers of the previous competitions in Sapporo (2017), Alcalá (2016) and Banff (2015).Attending the competition in Sapporo, September 2017, was crucial to understand the dynamics of the competition and draft the desired outputs for 2018.Thus, the final agreement did not only cover the competition days but also all the key aspects to organize the competition such as the venue mapping (see Section III) and the off-site data collection.Although the shopping mall managers were open to host the competition, they imposed the tight constraint of not disturbing the regular activities and daily-life in the shopping mall.• Planning in advance.The competition challenges and routes must be closed before signing the agreement.In particular, the organizers defined all the competition challenges and scheduled the data collection for the offsite tracks in November 2017.In December 2017, all the waypoints and the final evaluation routes were also defined.Therefore, the evaluation paths were already known by the competition organizers at the end of 2017.As it has been said, there was little margin to, for instance, add a new competition track or extend an existing one after signing the agreement with the shopping mall.• Careful site surveying and mapping.The accurate mapping of the competition venue is needed to provide the ground truth references within the ISO 18305 parameters.This procedure lasted 3 months, including onsite surveying and off-site post-processing.The venue and all the waypoints involved in the evaluation of the four tracks were accurately mapped.Lidar measurements were done the third week of January 2018 and the georeferenced waypoints were provided to Track chairs the first week of April 2018.• Thorough checking of competition data.The data sets for the off-site tracks were provided to competitors in the last week of April 2018.The data sets release included the technical documentation and supplementary files in terms of georeferenced maps.Although the raw data sets were collected and cross-checked in January 2018, it took two weeks to generate the definitive anonymized geo-referenced data sets and the supporting maps.It is of importance to remark that data sets cross-checking has been of paramount importance in the competition.An error in one of the reference files or the evaluation trajectory might be enough for withdrawing a competition track.For the data sets collection, two people manually logged the key sequence of waypoints for all the log files separately.In case of inconsistency between the two logs, the wrong manual log was fixed if possible or, otherwise, the log file was removed.For the process of dataset anonymization and georeferenciation, also two people checked that the resulting logfiles with anonymized WiFi data and georeferenced landmarks were coherent with the manual logs and the raw log files collected in January.• Screening of contestants by previous analysis of their methodological approach to indoor positioning.The deadline to send the technical descriptions was set to the second week of July.All proposals were evaluated by two independent track chairs, who analyzed the proposal feasibility.Although 34 teams participated in the competition, we reviewed more than 40 proposals.• Commitment to the competing teams.During the Setup day, three chairs were in the shopping mall to act as the main interface with the shopping mall and to give some clarifications to the competitors about the competition rules.During the competition day, the local organizing team (10 people) plus 2 external people were engaged in the competition organization since 6 AM.They covered multiple roles in the competition, which also include the installation of ground stickers that defined the competition courses early in the morning of the competition day.Thanks to them, the competition ran smoothly.
Regarding the on-site tracks, one team complained that the Non-camera based positioning challenge (Track 2), should have been split into two different challenges.They claim that Smartphone-based solutions had fewer opportunities to win and, therefore, they should have been evaluated independently.The organizing team defined the four challenges many months before receiving the description of the competition teams.Although one of the off-site challenges dealt with Smartphone-based positioning, the organizing team decided not to forbid that kind of system in the Non-camera based VOLUME 4, 2016 positioning challenge.This was reported to the organizers of the competition in 2019, who finally included an onsite smartphone-based positioning challenge.Although all scenarios cannot be covered in the same competition edition, they can be considered if the event is run annually like this one.
One of the competitors claimed that the competition score only reflected the positioning accuracy without balancing it with the resources required.Previous editions of the EvAAL competition indeed considered the accuracy, installation complexity, user's acceptance, availability and integrability into AAL systems in the evaluation criteria [1].However, any criteria will be subject to criticism due to the subjective selection of individual assessment criteria or their weighting in the final score.Therefore, the organizers selected an objective metric, the positioning error, as the main evaluation criterion having in mind that the competition is about indoor positioning and navigation.A solution to the issue raised by the competitors would be to organize many multiple tracks covering all the different aspects of the indoor positioning systems, but this approach was not possible due to limited resources that prevented the organizers to organize more than two on-site competition tracks.It is important to highlight, that the results of the competition and this document, which contains the description of some competing systems, are publicly available.Therefore, the competitors can argue and discuss their solutions considering additional parameters like complexity.
Some teams reported that this event allowed them to establish new collaborations with other participants.Some collaborations have been materialized into research visits, possible joint participation in the next editions and detailed discussions about the different strategies followed in the competition.Although the spirit of a competition is to compete, the organizing team considers that its main aim is to bring together the best research groups and encourage them to discuss their proposals.This is a very practical and less formal scenario to exchange knowledge and, above all, learn from the experience.Some off-site competitors realized that the teams reporting the best results in Track 3 used data from the magnetometer.It seems that the magnetic data can be really useful to refine the estimations provided by, for instance, Wi-Fi fingerprinting.Most of the teams that used Wi-Fi and inertial data are considering to add this additional source of information to improve their systems.Thus, some teams gained some empirical experience because of participating in the competition and they will use this expertise to improve their approaches for the next editions.
To sum up, organizing a competition in a real-life environment: a shopping mall has been a challenging task.It supposed an additional workload that involved more than 12 people, including the organizing research team and collaborators from other universities and research centers, to cover all aspects: accurately mapping the shopping mall, preparing four independent challenges, recording data for the off-site tracks, organizing a two-day event in a shopping mall without disturbing the customers.The on-site competitors had a challenging scenario to survey, in just one day, during the regular opening hours of the shopping mall without deploying any infrastructure to support localization, whereas the off-site competitors had to deal with some precollected data to provide the estimated positions of the blind evaluation path.The evaluation in the shopping mall, either on-site or off-site, also supposed an additional workload to the teams that participated in the competition.Although the workload was huge on both sides, it deserved any invested minute on the organization.All the competitors thanked the organization team for arranging the competition in such a challenging environment.The proposed evaluation was valuable to demonstrate that their solutions can work in a real-life scenario, which in turn is what the market is demanding.

VI. CONCLUSION
For the first time, the IPIN 2018 international indoor positioning competition took place in a real-life challenging space: the Atlantis shopping mall in Nantes (France).Four challenges (2 in real-time, 2 in post-processing mode) were prepared for the 49 registered worldwide competition teams.A survey combining differential GNSS, theodolite and mobile mapping 2D scanners enabled to map 9,000 m 2 and to compute the coordinates of 180 ground points spread over the 3 levels of the commercial center, with a 3D accuracy better than 10 cm.On September 22, 2018, all stickers, marking the way-points and used to define the paths to be followed by the competitors, were re-positioned on the ground thanks to the surveying and logging strategy followed.
This paper summarizes the main contributions presented in the competition and the details of the full results for comparison purposes.The ambition of this paper is also to draw lessons learned by the organizing committee but also by the competitors to improve the organization of future indoor positioning and indoor navigation competitions.
The most important conclusion is that the IPIN competition is not just an event where the participants compete.The main added value of the competition is the postevaluation discussion.The informal talks about indoor positioning between the competitors are an excellent way to exchange knowledge.On-site competitors witnessed many different novel approaches and technologies for indoor positioning running in a challenging scenario.Off-site competitors learned the benefit of processing magnetic data for smartphone-based applications.While many indoor location technologies rely on an infrastructure of beacons deployed in buildings, the competition results have shown that it is possible to achieve high accuracy positioning with selfcontained technologies.Finally, the competition has enabled some collaborations between different research teams, which was one of the main objectives of the competition.In general, the competition was not just a two-day event to compete for a prize, but a forum to discuss, establish new collaborations and learn.

FIGURE 2 .
FIGURE 2. Smartphone data collection in the shopping mall for Track 3 (left part).Foot-mounted PERSY device used in Track 4, which includes a STIM 300 inertial unit, an HMC5983 magnetometer and a NEO-M8T GNSS receiver (right part).

FIGURE 3 .
FIGURE 3. Marking the ground points to map the 180 waypoints constituting the pedestrian tracks during the shopping mall survey (left part) and during the on-site competition day (right part).

FIGURE 5 .
FIGURE 5. Network of GNSS and theodolite reference points used to georeference the shopping mall plans and perform global adjustment of indoor waypoints.

FIGURE 8 .
FIGURE 8. Detection of light-sources on a binary image and their extracted center mass points.

FIGURE 9 .
FIGURE 9.The concept of using ceiling lights as landmarks for indoor navigation.

FIGURE 10 .
FIGURE 10.Evolution of the particles in the localization Filter.In white: accessible zones (A) and in black and brown: inaccessible ones (B).At the init.state, the particles are uniformly distributed (left map).Using the short motion vector, the particles get organized in few clusters (middle map).Then the particles converged to a single position cluster (right map).

VOLUME 4, 2016
This work is licensed under a Creative Commons Attribution 4.0 License.For more information, see https://creativecommons.org/licenses/by/4.0/.This article has been accepted for publication in a future issue of this journal, but has not been fully edited.Content may change prior to final publication.Citation information: DOI 10.1109/ACCESS.2019.2944389,IEEE Access V. Renaudin et al.: Evaluating Indoor Positioning Systems in a Shopping Mall: The Lessons Learned from the IPIN 2018 Competition

VOLUME 4, 2016
This work is licensed under a Creative Commons Attribution 4.0 License.For more information, see https://creativecommons.org/licenses/by/4.0/.This article has been accepted for publication in a future issue of this journal, but has not been fully edited.Content may change prior to final publication.Citation information: DOI 10.1109/ACCESS.2019.2944389,IEEE Access V. Renaudin et al.: Evaluating Indoor Positioning Systems in a Shopping Mall: The Lessons Learned from the IPIN 2018 Competition

FIGURE 1 .
FIGURE 1.The technical route of the algorithm.

FIGURE 14 .
FIGURE 14.The technical route of the algorithm.

FIGURE 2 .FIGURE 15 .
FIGURE 2. The PDR algorithm for calculating the pedestrian walking trajectory and position.

FIGURE 3 .FIGURE 16 .
FIGURE 3. The Kalman filter is designed to combine different calculation results of these algorithms.

VOLUME 4 ,
2016 10.1109/ACCESS.2019.2944389,IEEE Access V. Renaudin et al.: Evaluating Indoor Positioning Systems in a Shopping Mall: The Lessons Learned from the IPIN 2018 Competition FIGURE 20.Floor plan modeled as polygon with details of the grid.

18 VOLUME 4
, 2016 This work is licensed under a Creative Commons Attribution 4.0 License.For more information, see https://creativecommons.org/licenses/by/4.0/.This article has been accepted for publication in a future issue of this journal, but has not been fully edited.Content may change prior to final publication.Citation information: DOI 10.1109/ACCESS.2019.2944389,IEEE Access V. Renaudin et al.: Evaluating Indoor Positioning Systems in a Shopping Mall: The Lessons Learned from the IPIN 2018 Competition

20 VOLUME 4 FIGURE 23 .
FIGURE 23.Concept of the INS/GNSS data fusion.The different parts of the INS are colored in blue and the augmentation of GNSS data is colored in yellow.

FIGURE 24 .
FIGURE 24.Compact subdivision of the human gait of the motion classifier during one step forward in four different motion states: Loading response, Midstance, Terminal stance and Swing[59]

FIGURE 25 .
FIGURE 25.Block diagram of the basic navigation algorithm

FIGURE 27 .
FIGURE 27.Flow chart of the moving trajectory estimation algorithm.
10.1109/ACCESS.2019.2944389,IEEE Access V. Renaudin et al.: Evaluating Indoor Positioning Systems in a Shopping Mall: The Lessons Learned the IPIN 2018 Competition

24 VOLUME 4
, 2016 This work is licensed under a Creative Commons Attribution 4.0 License.For more information, see https://creativecommons.org/licenses/by/4.0/.This article has been accepted for publication in a future issue of this journal, but has not been fully edited.Content may change prior to final publication.Citation information: DOI 10.1109/ACCESS.2019.2944389,IEEE Access V. Renaudin et al.: Evaluating Indoor Positioning Systems in a Shopping Mall: The Lessons Learned from the IPIN 2018 Competition 10.1109/ACCESS.2019.2944389,IEEE Access V. Renaudin et al.: Evaluating Indoor Positioning Systems in a Shopping Mall: The Lessons Learned from the IPIN 2018 Competition

TABLE 1 .
Comparison of the four challenge paths This work is licensed under a Creative Commons Attribution 4.0 License.For more information, see https://creativecommons.org/licenses/by/4.0/.This article has been accepted for publication in a future issue of this journal, but has not been fully edited.Content may change prior to final publication.Citation information: DOI 10.1109/ACCESS.2019.2944389,IEEE Access V. Renaudin et al.: Evaluating Indoor Positioning Systems in a Shopping Mall: The Lessons Learned from the IPIN 2018 Competition

•
Re-sampling: The process of choosing a new set of particles P from P according to the particles' weights.
10.1109/ACCESS.2019.2944389,IEEE Access V. Renaudin et al.: Evaluating Indoor Positioning Systems in a Shopping Mall: The Lessons Learned from the IPIN 2018 Competition 10.1109/ACCESS.2019.2944389,IEEE Access V. Renaudin et al.: Evaluating Indoor Positioning Systems in a Shopping Mall: The Lessons Learned from the IPIN 2018 Competition

TABLE 2 .
Route line plane error assessment results

)
VOLUME 4, 201610.1109/ACCESS.2019.2944389,IEEE Access V. Renaudin et al.: Evaluating Indoor Positioning Systems in a Shopping Mall: The Lessons Learned from the IPIN 2018 Competition

Table 3
shows the scores for all the four challenges.Additional metrics included in the ISO 18305 Standard are also reported in the table.VOLUME 4, 2016 10.1109/ACCESS.2019.2944389,IEEE Access V. Renaudin et al.: Evaluating Indoor Positioning Systems in a Shopping Mall: The Lessons Learned from the IPIN 2018 Competition

TABLE
Competition results for the four challenges based on the 75 th percentile (the competition main metric) and other well-known metrics of the positioning error as defined in eq.29Track Rank Team Name Mean RMS Median 75 th P. 90 th P.