Off-Line Evaluation of Indoor Positioning Systems in Different Scenarios: The Experiences From IPIN 2020 Competition

—Every year, for ten years now, the IPIN competition has aimed at evaluating real-world indoor localisation systems by testing them in a realistic environment, with realistic movement, using the EvAAL framework. The competition provided a unique overview of the state-of-the-art of systems, technologies, and methods for indoor positioning and navigation purposes.Throughfair comparisonof the performance achieved by each system, the competition was able to identify the most promising approaches and to pinpoint the most critical working conditions. In 2020, the competition included 5 diverse off-site off-site Tracks, each resembling real use cases and challenges for indoor positioning. The results in terms of participation and accuracy of the proposed systems have been encouraging. The best performing competitors obtained a third quartile of error of 1 m for the Smartphone Track and 0.5 m for the Foot-mounted IMU Track. While not running on physical systems, but only as algorithms, these results represent impressive achievements.


I. INTRODUCTION
T HE International conference on Indoor Positioning and Indoor Navigation (IPIN), born in 2010, has been a reference for researchers and practitioners interested in systems, methods, techniques and technologies for indoor positioning and indoor navigation.In fact, estimating the location of a mobile target still represents a challenging task in indoor environments.While solutions based on Global Navigation Satellite System (GNSS) are successfully used outdoor, pinpointing the location of an indoor target requires the adoption of technologies that most often cannot exploit satellites because indoor obstacles, walls and, most of all, ceilings are all factors that significantly reduce the strength of satellite signals.Indoor localisation systems, be they targeted at personal navigation or other purposes, heavily rely on the use of a wide variety of sensors.This is in sharp contrast with outdoor localisation, which relies only on GNSS radio signals, at least as far as consumer-grade applications are concerned.
Since its inception, IPIN's core topics have been low-level hardware and software techniques for positioning and navigation.In the last few years a growing interest has been observed in topics regarding system evaluation, standardisation and interoperability.In fact, reaching a wide consensus on the evaluation metrics for these systems is a fundamental step towards filling the gap between prototypes and commercial systems.In this paper we are particularly interested in testing and evaluation of systems and, as a showcase, we fully describe the IPIN 2020 competition.While the competition usually benefits from the attendance of the congress, in 2020 the competition was a solo event which nonetheless attracted 95 attendees to the final event, which was held online.
Past editions of the IPIN competitions were organised by hosting two different kinds of Tracks, namely on-site and off-site.In the on-site Tracks, competitors demonstrate their system by performing an assigned test in a given place.An actor carries the competing system while walking in and between multi-floor buildings.The system shall provide position estimates in real time using local data processing on opportunistic signals, without any ad hoc infrastructure.In off-sites Tracks competitors calibrate their algorithms in advance using a ground-truth reference database provided by the committee, and compete using new unreferenced data.Due to worldwide travel restrictions, the 2020 competition only hosted off-site tracks for active indoor positioning systems.
This paper contains organisational aspects and highlights the choices taken by the organisers.The core part of the paper is the description of the competing systems.This edition provided five off-site Tracks: Smartphone, Foot-mounted IMU, xDR in manufacturing, On-vehicle smartphone and Channel impulse response.Each Track is explained in a dedicated section which also contains contributions and system descriptions authored by the competitors.As a follow-up to [1], [2], this work provides a unique overview on the state of the art of systems, technologies and methods for indoor positioning and navigation purpose.Through a fair comparison, the performance achieved by each system in a real-world scenario helps understanding which are the most promising approaches, under which working conditions.Comparison is performed according to the Evaluating Ambient Assisted Living (EvAAL) framework [3].
The paper is structured as follows.Section II summarises the history of the IPIN competition and highlights possible future directions for the next editions.Section III is an overview of the five Tracks and their commonalities, which are founded on the EvAAL framework.Sections IV to VIII report the characteristics and final results for each Track and the detailed descriptions of most competing systems.Section IX is an attempt at identifying lessons to be learned from the practical experience of competitors: even if the competition was off-site, the algorithms used were stressed in a challenging and competitive environment, which offered insight to both competitors and attendees.

II. PAST, PRESENT AND FUTURE DIRECTIONS
Research in the area of indoor positioning and navigation in the last decade has elicited a strong interest from both academic and industrial communities.We expect indoor Location-based services (LBS) to experience significant growth and evolution and to be commonly available on commercial devices in the future Although impressive advances in the field of algorithms for indoor localisation and tracking have been achieved, evaluation frameworks are missing.Under this respect, the EvAAL framework was a pioneer initiative devoted to compare, with a rigorous methodology, the performance of indoor localisation systems in real-world, non-trivial settings.Here we summarise the 10-year-long journey of the EvAAL framework, from the first EvAAL competition in 2011 to the recent IPIN 2020 competition, and we give a look at the next edition of IPIN scheduled for late November 2021.
The EvAAL framework has been designed to test and compare the performance of indoor localisation systems, following a rigorous approach.It consists of four core criteria plus four extended criteria, the latter being desirable ones which should be applied as far as possible [3].The core criteria, which are necessary to define a competition as conformant to the EvAAL framework, are: 1) Natural movement of an actor: an actor walks with natural speed and attitude.2) Realistic environment: the walking path is set in a realistic setting; EvAAL competitions were done in a living lab, IPIN competitions in wider settings, like a congress centre, a university building, a shopping mall.3) Realistic measurement resolution: final error measurements below 50 cm in space and 0.5 s in time should be considered as null, when indoor people's movement are considered; when the actor walks, the test should be considered adequate if his/her time and space errors when passing on the test points are not greater that the above figures, which is easy for a trained person.4) Third quartile of point Euclidean error: the accuracy score is based on the third quartile of the point error.Applicability of extended criteria to IPIN 2020 is discussed in Section III.
Table I is an overview of the size of past competitions.While the IPIN competitions aim to compare systems based only on their accuracy performance, the early EvAAL editions were characterised by a richer set of goals, including the deployment complexity of the solution; the time required to calibrate and configure the system; the impact of the system in terms of the end-user's perception.These indicators were mainly driven by the Ambient Assisted Living (AAL) application scenarios to which EvAAL was inspired [4].

TABLE I NUMBER OF TRACKS AND NUMBER OF COMPETITORS FOR ON-SITE AND OFF-SITE INDOOR LOCALISATION COMPETITION TRACKS
In 2014, the EvAAL competition met the IEEE IPIN conference, giving birth to the IPIN competition [5].Such partnership was the result of two complementary communities: on the one hand, experience from the EvAAL competitions provided a well-established evaluation framework; on the other hand the IPIN provided a vibrant community composed of academic and industrial attendants who every year share advances in the area of indoor navigation and positioning.The birth of the IPIN competition series extended the range of potential competitors.Indeed, the IPIN competition has seen a consistent increased of the number of competition Tracks, each of which focused on specific constraints and objectives, as shown in Table I.Tracks are split between on-site and off-site.The on-site Tracks take place during the IPIN conference and competitors do a live test of their solutions.Off-site Tracks are performed remotely.For the latter ones, competitors are required to test their solution by following rules and data sets provided by the organisers.
During the last 7 IPIN competition editions, competitors have had the opportunity to test their systems in shopping malls, conference halls, university campus and large research centres.Such a variety of locations is the distinguishing feature of IPIN competitions with respect to similar initiatives.In fact, the confluence of EvAAL into the IPIN conference refined the methodology adopted to assess the performance, adding the following characteristics: no additional instrumentation allowed, non-overlapping competition Tracks, highly representative competition areas, easy-to-understand measurement statistics to define the final ranking of the tested systems.
Appreciation of this format by competitors (both from academy and industry) and sponsors is reflected in the consistently growing attendance to the competition.
So far so good, but what's next for the IPIN competition?Organisers are looking at two growing trends: • the increasing performance and diffusion of sensing units available with commercial devices; • the wide adoption of learning methodologies with a never-seen-before statistical power.As far as sensing is concerned, new short-range Radio-Frequency (RF) technologies such as Wi-Fi Time-of-Flight (TOF) measurements, Ultra-Wide Band (UWB) and Bluetooth 5.0 are the next obvious target to include in testing by augmenting the existing Tracks or creating new ones.In the future, medium-and long-range RF technologies 5G and 6G may become drivers for localisation technology, but currently it is not easy to set up a representative testbed: telecommunication providers might play a crucial role for indoor localisation; the IPIN competition is open to testing and experiencing such disrupting technologies.
As far as the increasing pervasiveness of machine learning is concerned, our prospect is to support such evolution by offering always-more challenging data sets to the competitors, in order to assess the performance of their systems, as it has been done with the off-site Tracks.Under this respect, we consider heterogeneity as one of the most challenging properties of such data sets.Heterogeneity refers to the different nature of data that can be simultaneously analysed, to improve the performance of ML-based algorithms.Fingerprint data sets, based on of Wi-Fi Received Signal Strength Indicator (RSSI) readings, can be enriched with context information derived from Bluetooth beacons, environmental or physiological sensor readings, giving rise to unexpected possible correlations.In turn, such data sets can be used as non-structured inputs to multi-layer neural networks (e.g.Recurrent Neural Network (RNN) based on Long Short-Term Memory (LSTM) and Gated Recurrent Units (GRU) layers) to solve classification and regression problems applied to indoor localisation.This trend has been in place in both on-site and off-site Tracks in the last years, and it is going to continue.
Another interesting topic where the IPIN competition could promote new challenges is the adoption of a more accurate and meaningful metric for computing the positioning error.In fact, one of the objectives of IPIN is to define standard procedures for evaluating positioning systems, in an effort to improve over the recent ISO/IEC 18305 standard [6], [7].Discussion is underway about using an alternative or additional criterion for computing the error, that is, the distance from each reference point in the ground truth to the position estimated by the competing system.Currently, the IPIN competition series defines the point error as the horizontal distance plus a fixed penalty of 15 m per each wrong floor.Now times appear to be mature for the adoption of a "real-world" distance, that is, the length of the path that a person would need to travel from the reference point to the estimated point.This is the same as the Euclidean distance if the two points are in line of sight, that is on the same floor and in the same room, but can be very different otherwise.A complete discussion on the benefits of this new method and of the possible algorithms to use, complete with code, is available at [8].
A final consideration about the future of the IPIN competition refers to the integration of multiple indoor localisation systems.More specifically, we envision a future where different indoor localisation services coexist in the same area.Such systems will require to be integrated and orchestrated so that to reproduce, as much as possible, the well-assessed user experience of navigation in outdoor environments [9].We consider two key challenges: • to standardise Application Programming Interfaces (APIs) designed to discover, access and use an indoor localisation systems with a commercial device; • to regulate the privacy consents asked of end-users in order to provide location-based services in accordance with the EU General Data Protection Regulation (GDPR) regulation framework.

III. ORGANISING AN OFFSITE COMPETITION WITH MULTIPLE TRACKS
In the 2014-2019 editions the systems competing in on-site Tracks were tested during the IPIN conference, in the same or a nearby site.This way, competitors were able to both compete with their system and attend the conference, and conference attendees could come and look at how the competition was done.Since the conference was cancelled in 2020, only off-site Tracks were organised for this edition, under the supervision of competition chairs Francesco Potortì and Sangjoon Park.
The institutions involved were the Institute of Information Science and Technologies (ISTI) of the National Research Council (CNR, IT), UBIK Geospatial Solutions S.L. (ES), the GEOTEC laboratory of Universitat Jaume I (UJI, ES), Consejo Superior de Investigaciones Científicas (CSIC, ES), the IN3 of Universitat Oberta de Catalunya (UOC, ES) the GEOLOC Team, University Eiffel (FR), the National Institute of Advanced Industrial Science and Technology (AIST, JP), University of Tsukuba (JP), Aerospace Information Research Institute, (CAS, CH), IIS Fraunhofer (DE) and the Electronics and Telecommunications Research Institute (ETRI, KR).

A. Preparing the Competition Areas
In contrast with the two previous editions where most on-site and off-site Tracks took place in the same large area (a shopping mall in 2018 [1] and a research centre in 2019 [2]), in 2020 travel restrictions made it impractical to gather together and take measurements in the same place, so all Tracks were independent.
The set of evaluation scenarios cover a university library building, the shopping mall from IPIN 2018 competition [1], a manufacturing site, road-based tracks with different satellite view conditions (including indoors) and an environment resembling an industrial setting.
All the Tracks complied with the EvAAL framework [3] by adopting its four distinguishing core criteria (described in Section II): 1) Natural movement of an actor 2) Realistic environment 3) Realistic measurement resolution 4) Third quartile of point Euclidean error Additionally, all the Tracks were compliant with most of the extended criteria defined by the EvAAL framework, as detailed below.
1) Secret Path: The final path is disclosed immediately before the test starts, and only to the competitor whose system is under test.This prevents competitors to design systems exploiting specific features of the path.This criterion is always respected in all Tracks given the way the off-site competition Tracks are set up: competitors are provided with training sets, ground truth and, in some Tracks, a map.When they have finished tuning their systems, they ask the organisers for a path without ground truth, and submit their estimate.The ground truth is published only after the competition is finished.
2) Independent Actor: The actor is an agent not trained to use the localisation system.This criterion is always respected, given the way the off-site Tracks are set up.
3) Independent Logging System: The competitor system estimates the position at a rate of twice per second. . . .This criterion is respected or exceeded in all Tracks.
. . .and sends the estimates on a radio network provided by the committee.This prevents any malicious actions from the competitors.The source code of the logging system is publicly available.This criterion is not respected, because the competitors may retry and further tune their systems while trying to guess the correct ground truth.To avoid this, the committee should ask the competitors to provide their code, and run it locally in a real-time fashion, or provide a real-time APIs.This is feasible, in principle, but would require a non-trivial software infrastructure to be in place, and a non-trivial additional effort from the competitors to comply with it.
4) Identical Path and Timing: The actor walks along the same identical path with the same identical timing for all competitors, within time and space errors smaller than the above defined measurement resolution.This is a natural consequence of the fact that the same data are provided to all competitors.

B. Competition Results
For each submitted trial, the error was computed by comparing the estimated coordinates with the ground truth, that is, reference coordinates of the key points marked on the ground along the path.This metric combines the floor detection accuracy and the horizontal positioning error. where • P R is the vector with the ground truth horizontal (2-D) coordinates • P E is the vector with the horizontal coordinates estimated by the competitors • P R − P E is the horizontal error, and it is computed as the Euclidean distance between the ground truth and the estimated position provided by the competitor in the 2D space.
• p is the base floor estimation error penalty and is set to 15 m.
is the absolute difference between the actual floor number and the estimated one.The point error ε is computed for all key points marked on the ground that define the path of a specific challenge.The "accuracy score" s is given by the third quartile of ε: The team with the lowest score wins the challenge.Note that each competitor had the opportunity to submit the results for multiple trials.Table II shows the scores for all the five Tracks.Some additional metrics included in the ISO/IEC 18305 standard are also reported in the table.Fig. 1 depicts the cumulative distributions of the accuracy score s for the winners and runners-up of the five Tracks.

IV. TRACK 3: SMARTPHONE A. Track Description
The goal of Track 3 is to evaluate the performance of different integrated navigation solutions based on a regular smartphone sensor fusion (magnetometer, barometer, wireless communications, Attitude and Heading Reference System (AHRS) or micro-electro-mechanical systems (MEMS), among others) in an off-site context.As done in the 2016-2019 editions [1], [2], [10], [11], the same data collection strategy and evaluation procedure has been followed.The competition data set was collected by the same actor using a Samsung Galaxy A5 2017 (SM-A520F) phone with Android 8.0.Despite being three years old, this model has the advantage of having being used in Track 3 for 2018 and 2019 competitions.The main features of the embedded sensors, including the maximum sampling frequency, are summarised in Table III.
As in the previous competitions, we have used the Android app "GetSensorData" [12] to record and store the smartphone sensors data into a single text file, i.e. into a logfile.The data set is split into three subsets, namely training, validation and evaluation: • The first set is devoted to calibration purposes and covers most of the evaluation area; it contains 18 short single-floor tracks (collected 4 times each), 4 long trajectories across bookshelves and 2 floor transition tracks.We placed key points at every relevant location, i.e. initial/final locations, significant turns in the tracks and the last step to arrive at a new floor.A total of 78 training logfiles were provided to competitors.• The second set is devoted to validation purposes, allowing competitors to have an initial assessment of the positioning system, and contains 13 multi-floor long tracks.The number of key points is arbitrary and significantly lower than in the training set.A total of 13 validation logfiles were provided to competitors.• The last set is devoted to evaluation purposes, allowing competitors to have an independent evaluation without ground truth data, and contains just 1 multi-floor very long track.In contrast to the systematic data collection done in the training files, the evaluation logfile included realistic movements (e.g.simulating a user that was messaging or attending a phone call) and stops.Only 1 unlabelled logfile was provided to competitors.We set the maximum allowed sampling frequency in "Get-SensorData" for all sensors to record as much as possible data.Additionally, the smartphone was not connected to any cellular or Wi-Fi network as, for instance, the Wi-Fi sampling frequency significantly drops when the phone is connected to a Wi-Fi network.The logfiles and supplementary materials are available in [13].This package complements the ones from the previous editions [14]- [17].

B. Competition Area
The environment selected for Track 3 is a modern multi-storey library building located at Universitat Jaume I (Castellón, Spain) and includes a small outdoor area near the main entry.This environment covers the use case for a smartphone application guiding students and staff to find the location of a book.
Before collecting data, the library building was visually inspected to identify the most challenging parts where competitors could find it difficult to obtain accurate positioning.We finally selected the main hall entry (≈300 m 2 ), the second floor (≈1000 m 2 ), the third floor (≈900 m 2 ), the floating fourth floor (≈200 m 2 ) and the fifth floor (≈700 m 2 ) to collect data.We discarded common areas on the first floor and zones with restricted access.The library is composed by two interconnected blocks, we mainly collected data in the first block, except for the fifth floor, where part of the second block was finally surveyed.
For the evaluation path, we considered a walk done by a student that was doing some homework in the library.The student starts sitting in his/her work place (on the third floor), the student stands up and looks for a book, attends a phone call -despite it not being allowed-, comes back to the main workplace and stops for a while.Then, the student needs additional materials that is on the fifth floor, and goes directly there.The book is not in the place it was supposed to be, and the student asks a mate.The book seems to be in the new bookshelves located in the same floor but in the second block (left side in Fig. 2).On the way to the second block, the student meets a friend in the floating fourth floor (which was not mapped).Our student gets the book, returns to the work place, but the computer and other materials are gone.The student has not realised being instead on the second floor and starts to look around desperately.The student goes out to notify the security staff about this event.When the student goes back to the workplace on the third floor, he/she realises everything is there and sits to continue working after 20 minutes walking.The path goes through 82 key points for a total length of approximately 1000 m.
We have used geo-referenced indoor maps (ArcGIS engine) and the ArcMap tool to calibrate all the reference points used in the data set.We performed on-site local measurements using a laser distance meter with respect to representative points, such as walls, pillars and doors, which were already well represented in the indoor maps.The inaccuracies deriving from this procedure might be considered irrelevant as all the information and indoor maps are provided to all competitors.C. Indoor Positioning Solutions Provided by Competitors 1) Team WHU-FIVE: Just like many indoor positioning systems, the WHU-Five system includes two stages, the training stage and the testing stage.In the training stage, the system attempts to build fingerprints, train the models and extract some information about the positioning environment.In the testing stage, it deals with the testing data to form the final trajectory, leveraging the trained information and fusion algorithm.In order to build the feature maps, the position of every sampled signal features must be known.The reference points are used in the training sets to optimize the Pedestrian Dead Reckoning (PDR) algorithm to obtain a highly accurate trajectory estimations.The model between PDR and reference points can then be build.The model is optimized with the Levenberg-Marquardt (LMA) optimal algorithm.With the optimal PDR, the feature maps for Wi-Fi and geomagnetic can be build.Also, the book-shelf layouts can be extracted.As for the step length model, a leverage linear regression is used to train the model parameters.For the stair-step number between each layer, the average steps of each stair on the stair training data set is used.The motion recognition is performed by training a multi-layer perceptron neural network model.The time domain and frequency domain characteristics are extracted from the original data of the Inertial Measurement Unit (IMU) and barometer sensors, and then are input to the neural network.The output of neural network is the motion types set.
b) Testing stage: In the testing stage, first, a Wi-Fi fingerprint positioning is used to find out the initial 3-dimensional position.Then, PDR is fused with the building map and geomagnetic fingerprint positioning result to estimate the 2-dimensional trajectory.Meanwhile the motion recognition  result is used to revise some error estimation of PDR.Lastly, the motion on stairs is used to estimate the layer id.Combined with the layer id, the 2-dimensional trajectory can be built up to 3-dimensional trajectory.
For the final testing trajectory estimation, the IMU is used to provide the original PDR.The recognized in-situ steering motion is used to weed out the corresponding steps and the phone-holding posture is used to revise the heading of PDR.Then the revised PDR is fused with the building map and book-shelf layout in particle filtering algorithm to obtain further trajectory estimation.The geomagnetic fingerprint matching positioning result is then fused with the previously estimated trajectory in Kalman Filter (KF) algorithm to obtain the final 2-dimensional trajectory estimates.Combining the layer estimation result and the 2-dimensional estimates, the final 3-dimensional trajectory is obtained.The algorithm of testing stage of the system as shown in Fig. 4.
2) Team IOT2US: IOT2US team system includes four main stages: 1) the floor and region decisioning based on Wi-Fi; 2) the mobility mode detection; 3) the landmark detection; and 4) the PDR algorithm and information fusion.Six types of sensor data were fused in the whole processing of the track reconstruction and each part involved some of them, as can be seen in Fig. 5.In the following paragraphs every stage is explained in detail.
a) Wi-Fi: According to the training data, there are five floors involved in this Track.To determine the floor information and get the rough location of the user, Wi-Fi Received Signal Strength (RSS) information is used to decide the floor and region.Here region is defined as the area that each training logfile covers.
There are two phases in the Wi-Fi process.The RSS fingerprints which contains time, floor, region and key-value pair of MAC and RSS, are built from the training data set during the offline phase.The RSS fingerprint can be represented as a vector of (ti me, f loor, region, mac, rss).And during the online phase, the Wi-Fi RSS information in the evaluation logfile is compared with the RSS fingerprints to compute the most suitable location.More specifically, there are two steps in the online phase.
First, the floor and the region of estimation of the point are determined with respect to the wireless Access Point (AP) availability.A coefficient (λ) is defined to indicate the possibility to estimate a point appearing in the region to which the RSS fingerprint belongs, where, the n e is the number of APs detected at the estimation point, the n r is the number of APs detected at the RSS fingerprint.The n is the number of APs detected at the estimation point and the RSS fingerprint at the same time.This coefficient λ is calculated for every RSS fingerprint that belongs to the same region.The region with the minimum sum λ is the estimated region to which the estimation point belongs to.Then the floor based on this estimated region can be obtained.Second, to get the rough location of the estimation point, the RSS information is compared against all RSS fingerprints.The Euclidean distance in the signal space between the estimation point and every RSS fingerprint is calculated as: where, the rss i e and rss i r are the RSS values of the i -th AP detected at the estimation point and the RSS fingerprint, respectively.
b) Movement modes recognition: Different modes of mobility can be detected using machine learning or deep learning algorithms using multi-sensors data.For this Track, four categories of motion modes were introduced: normal walking, turning, climbing (stairs), descending (stairs).The process chain mainly includes data segmentation, labelling, feature extraction and classification.In IOT2US team system, accelerometer, gyroscope, magnetic field, and pressure are used for motion modes classification.Some statistical characteristics (e.g., mean, max, derivative) of these time series in time-domain are extracted as features.Decision tree and Support Vector Machine (SVM) are investigated to classify these motion modes.
c) Landmark detection: Map information is one of the most important clues that can help to correct the trajectory.Traditional map matching trends to make use of the structure of the rooms, corridors and tunnels to restrict the trajectory.However, this request too many details of the map and sometimes to measure the building structure in detail is a heavy workload.Hence, some landmarks are identified that activities can only happen at certain places as Correction Reference Points (CRP) to correct the trajectory.Through analyzing the relationship between the real environment and sensors data, three types of CRP can be identified, which are determined by Barometer, Ambient light sensor, and door-crossing activity: 1) the activity of climbing and descending stairs can only happen at stairs, and hence, with a rough location using Wi-Fi can give at least two CRPs.
2) Light intensity measured by Ambient light sensor also has relationship with activities.As an example of walking across bookshelf activity, as shown in Fig. 6, the activity of walking out of the bookshelf and turning backing is recorded as a peak in light intensity.The light determined CRPs also can be determined at place of crossing doors, enter/leave the building and approaching to a window.3) The third type of CRP determined by door-crossing is similar with the first one that requires recognize the door-crossing activity and Wi-Fi to find a rough location to determine the position of a door.

d) PDR and Information Fusion:
Step counting, step/stride length estimation and heading determination are three crucial processes for PDR positioning systems.Peak detection [18] is used to count steps, the Weinberg method [19] to estimate step length and complementary filter [20] for heading determination.To overcome this the traditional PDR algorithm issue of error accumulation over elapsed time, information is fused, including movement modes, floors, regions and landmarks.First, as the sensor data may show distinctive characteristics when a user performs different activities, movement modes are used to finely tune the parameters of step counting and step length estimation.For example, when a user is climbing or descending stairs, the parameters of step counting and step length estimation algorithms is accordingly adjusted to improve their performance.Second, before calculating locations using PDR, Wi-Fi is used to determine the absolute floor and the region information where the user roughly locates.This process assisted IOT2US to provide absolute location to help calculating the PDR trajectory and determining a CRP.And third, if the user is detected as reaching at a CRP, but the calculation result diverges from it, the heading is adjusted, step length and the previous track, to drive the trajectory back to the CRP.
3) Team XMU_ATR: XMU_ATR proposed XMU_PDR system.It is a multi-source indoor positioning system using information obtained from a Inertial Measurement Unit (IMU), Wi-Fi, magnetometer, barometer and indoor maps, jointly.Fig. 7 shows the framework of the proposed system.There are four main functional modules described as in the following paragraphs.
a) Inverted pendulum model based Pedestrian Dead Reckoning (PDR): A raw trajectory is obtained from the inertial data using an inverted pendulum model based PDR algorithm [21] by estimating every pedestrian step length and heading angle directly.The proposed system uses an inverted pendulum model to calculate the step length.To achieve 3-D positioning, the barometer is used to estimate the transition of floors.In the PDR system, heading errors are one of the main factors when estimating positions which will lead to a decrease in positioning accuracy over time, other information needs to be introduced to correct the trajectory.b) Wi-Fi fingerprints matching: Wi-Fi fingerprints matching is a reliable way to obtain absolute indoor position.Since the ground truth of some reference points in the training set is provided, the Wi-Fi Received Signal Strength (RSS) is extracted at the reference points as fingerprints to build up the fingerprints database.Weighted k-Nearest Neighbor (WKNN) algorithm is used to match the Wi-Fi Access Point (AP) between fingerprints database and the RSS from evaluation data.Since the initial point is unknown, the trajectory obtained by PDR can only be presented in a temporary navigation frame automatic defined by the dead-reckoning system.Therefore, the result of Wi-Fi positioning can be used to estimate the transformation relationship between the navigation frame and the geographic frame, including coordinate translation and rotation.

c) Magnetic fingerprints matching:
The indoor magnetic field can be treated as a time invariance distribution in spatiality.The accuracy of spatial resolution can achieve centimeter level in a small area.The magnetic fingerprints matching is used for trajectory refinement.Since the trajectory to be evaluated may have some overlap with the training set, the observations of magnetometer in the training set are also used as labeled fingerprints.A modified dynamic time warping algorithm is used in this part which can deal with the matching problem between two sequences with different directions.A matching threshold is set to decide whether the matching is successful.If there is a trajectory in the evaluation set match to part of trajectories from the training set, this track can be located on the map.
d) Map matching: To reduce positioning error accumulated from the noise of inertial observations, the map information is used for trajectory calibration.The optimal estimation under the map constraint especially the track at specific locations such as walls, doors, and stairs are realized.The floorplan is presented in the form of grids, and a loss function is defined to adjust trajectory actions referring to walls and some specified behavior patterns.
4) Naver Labs Europe (NLE) Team: Naver Labs Europe (NLE) Team system is based on extending the localization pipeline developed for IPIN 2019 [1] challenge with new components.The pipeline is a sensor fusion framework deploying smartphone inertial sensors, Wi-Fi measurements, magnetic field data and landmarks.The main components are described in the following paragraphs: a) Floor detection: Floor is detected using a standard k-Nearest Neighbor (KNN) classifier trained on barometer and Wi-Fi data.
b) Activity detection: User's activity (walking, standing, going up or down the stairs) is identified by applying spectral analysis on the accelerometer data and simple thresholds.
c) PDR: User steps are first identified by applying peaks detection to accelerometer data.Then, the acceleration features are extracted and a model is trained for the step length/speed in a given sliding window.Together with the orientation sensor, a first order approximation PDR of user's track is built.

d) Semi-supervised Variational Auto-Encoder (VAE) for Wi-Fi:
A radiomap is constructed from the Wi-Fi data provided in the training and the validation data sets.Recorded data provides a Wi-Fi scan reading every 4 seconds approximately, but without an exact position where this scan was taken.Using the inertial sensor data, the approximate position can be inferred by using the semi-supervised Variational Auto-Encoder (VAE) [22].
e) Magnetic field based localization: This component for indoor localization uses magnetic field data captured by the mobile phone.In indoor environment, magnetic anomalies are created by different ferromagnetic objects.To benefit from their presence, the state of the art landmark-based classification [23] is extended.Once the magnetometer captures changes of the Earth's magnetic field due to indoor magnetic anomalies, they are transformed in multi-variate times series.Temporal patterns are then converted in visual ones by using 1D convolutions with Recurrent plots, Gramian Angular Fields and Markov Transition Fields (see Fig. 8).This represents magnetic field data as image sequences and permits to deploy convolutional layers to associate magnetic patterns with particular places.A deep regression is trained on the user's position and combined convolutional and recurrent layers in the deep network [24].
f) Deep PDR: PDR is processed by applying deep learning.Acceleration and orientation sensor data streams are pre-processed and represented as 2D images analogously to the magnetic field data.CNN and RNN are used to extract underlying hidden correlations between different sensors and modalities to learn a model of user local displacement.This allows coping with sensor noise and replaces the manual feature extraction which is frequently a subject to data noise and sophisticated thresholding, including tuning to different pedestrian profiles, depending on gender, age, height etc.The deep PDR model is learned to predict relative (x, y) displacements.The relative displacement model is trained using the regression loss on available annotated data.
The deep PDR model is locally accurate but accumulates errors over time.This PDR drift is compensated by using global localization components based on Wi-Fi and magnetic field based localization.
g) Landmarks and pseudo labels: CNN/Deep Neural Network (DNN) models require large-scale training data.However, genuine ground truth annotations are sparse and available for a limited number of landmarks.On the other hand, raw sensor data are massively generated at a high rate.So, sensor data is annotated with pseudo labels and a large annotated set for training CNN/DNNs is generated.Pseudo labeling is based on simpler tasks of user walking and landmark detection and an interpolation of user's behaviour between the landmarks using the first order approximation PDR.
h) Prediction fusion and map projection: Relative predictions provided by deep PDR and absolute predictions provided by Wi-Fi and magnetic field data are combined using an Extended Kalman Filter (EKF).The output of the filter is then finetuned, by projecting it on the paths that were traversed while collecting training and validation sets, to make sure that the final result lies within the navigation space in the building.
5) Team UMinho: The UMinho team approach for the 2020 competition (Fig. 9) was based on a Particle Filter (PF) to fuse Wi-Fi fingerprinting positioning with motion, heading and atmospheric pressure data.In a calibration or initialization phase, the positioning system is prepared by creating a space model (floor plan) and a Wi-Fi radio map created using the training data.The radio map and the floor plan are then used in the Validation and Evaluation phases by the PF to estimate the trajectory using PDR obtained from motion and heading data.
a) Creating the Wi-Fi Radio Map: The Wi-Fi radio map was created using the provided Training data sets.To obtain a higher quality radio map, correction techniques are applied to the trajectories obtained through PDR (module 2).The correction approach uses the ground truth points included in the Training data sets (POSI) to correct the distance and heading for each training trajectory.The description of the training data sets specifies that the user travels along a straight path between two consecutive POSIs, making it possible to estimate the travel distance error and the heading error between two POSIs.The segment between the two POSIs is then corrected proportionally by re-doing the PDR between the two POSIs, adjusting each step length and heading value so that the estimated position at the second POSI matches the ground truth.This approach works for the Training trajectories, except the ones including stairs.The corrected trajectories from module 2 are fused with Wi-Fi, pressure and POSI data in module 3 to estimate the floor and (x, y) position for each Wi-Fi sample in the Training data set.The radio map is obtained from this process.
b) Floor plan integration: The floor plan is integrated by processing the provided bitmaps or vector images for each floor.Some processing is required to incorporate the floor plan into the PF (module 1).The first step is to remove some elements that may prevent the particles from moving freely between the existing spaces, such as doors represented in the floor plan.Then the floor plan is converted to binary format.

c) Estimating the final trajectory:
The final trajectory is estimated by processing the Evaluation data set which is provided with the same types of data available in the Training data sets, except for the ground truth POSIs.Module 4 is responsible for estimating the displacement using the accelerometer measurements.The movement displacement is assessed by an algorithm that estimates the user's steps and corresponding length (algorithm also applied in module 3).The displacement and heading information for the Evaluation trajectory are obtained from module 4, necessary to perform PDR, which is integrated into the PF (module 6).Wi-Fi and pressure samples are combined in module 5 to estimate the floor and the z coordinate for each Wi-Fi sample.The radio map, created in the initialization phase, is used to perform Wi-Fi fingerprinting and improve the floor estimation.These enhanced Wi-Fi samples are then fused with the displacement and heading samples in the PF.The PF (module 6), based on the solution presented in [25], performs sensor fusion of Wi-Fi fingerprinting with PDR (displacement and heading).Particles are created around the initial position, which is estimated using Wi-Fi fingerprinting.Particles states follow a PDR motion model considering noise in the heading and displacement.Particles' weights are updated based on Wi-Fi fingerprinting, using a distance function to convert the distance between the particle and the Wi-Fi position estimate into a weight.Higher weights are assigned to particles closer to the Wi-Fi position estimate.To reduce errors from Wi-Fi fingerprinting, a partial radio map is used, considering only Wi-Fi samples that are in the neighbourhood of the PF estimated position.Particles with lower weights, including those that hit walls or obstacles, are resampled based on the multinomial resampling method.The floor changes provided by module 5 allow the PF to adjust the motion model when a user changes floors, reducing the step length.The PF also performs adjustments when a floor transition is detected to ensure that all particles are moved into the current floor.The final pose is obtained from the particles' positions and headings weighted average.
6) Team Imec-WAVES: imec-WAVES team positioning system consists of 4 parts: Pedestrian Dead Reckoning, RSS fingerprinting, floor (transition) detection and PF.Fig. 10 shows a flowchart of the system.These steps are introduced in detail in the following paragraphs.
a) Pedestrian dead reckoning: The PDR algorithm fuses the data of accelerometers, gyroscopes and magnetometers to estimate the trajectory.It consists of step detection, heading estimation and step length estimation.Step detection and heading estimation are based on [26].For step length estimation, an adaptive Weinberg model is used [27].The phone carrying mode is determined for each step by a KNN-classifier.The features used are average and variance of both roll and pitch from the AHRS data during one step.The competition training data include one carrying mode: holding the smartphone in front of the body.However, the competition introduction document mentions realistic movements (e.g.phone call).Therefore, additional training data was created by the team where an actor walked for several minutes while pretending to make a phone call.If a phone call is detected in the evaluation data, the heading is flipped 180 • .
b) RSS fingerprinting: The radiomap is constructed by interpolating between the known coordinates in the training data [1], [2].The Euclidean distance metric is used to match RSS vectors in the validation/evaluation data with RSS vectors in the radio map.The metric can only be applied to a subset of RSS values for each vector, depending on the APs the vectors have in common.A penalty is added to the Euclidean distance for each AP that is not in the subset.This prevents a (false) good match when the RSS vectors have only a few APs in common but with similar RSS values.The weighted centroid of the three best matches is selected as the estimated position.
c) Floor (transition) detection: The barometer is used to estimate height changes.The average pressure during each detected step is converted to a height difference relative to the first step [28].If the absolute height change is more than 1.5 m within a window of 15 consecutive steps, these steps are labeled as 'on stairs'.However, the pressure changes with time regardless of the actual height difference.Therefore, if the time between two steps is more than 20 s, the height difference between these steps is removed to prevent false stair detections when standing still for a long time.The height difference during each sequence of 'on stairs' labels is then used to estimate the floor difference.The middle step of each 'on stairs' sequence is labeled as 'transition'.The barometer method can accurately detect transitions, but can have errors in the estimated floor differences because the true floor height is unknown.The visited floor itself can be determined by matching RSS measurements with the radiomap, but sometimes the wrong floor is matched for a short time interval.The Viterbi algorithm, which was used for the previous competition to perform localization [29], is now used to find the most likely sequence of visited floors by fusing these RSS matching and barometer methods.
d) Particle filtering: A PF is used to fuse the output of previous parts and to perform map matching.To enable the latter, the provided floor plan images are converted to XML files containing the locations of each wall, staircase and elevator.This is done with the WHIPP tool [30], [31].The locations of bookshelves in the evaluation environment are deduced from the training data and regarded as walls.The implemented PF is the Backtracking Particle Filter (BPF) [32].During each iteration, the BPF uses new information to update current and previous states.At initialization, thousands of particles are generated uniformly over the floor plan.The amount of particles is drastically reduced during the next iterations when the filter starts to converge.The length and heading of the detected steps are used to propagate the particles.Artificial Gaussian noise added to the heading depends on the detected carrying mode.The floor plan is used to remove all particles that crossed a wall during propagation, as this is physically not possible.If RSS measurements are available for the current step, the position is estimated and a Gaussian curve is used to weigh the particles based on their distance to the estimated position.If the step is labeled as 'on stairs', the weights of particles outside of staircases are decreased.If the step is labeled as 'transition', the next floor plan and radiomap are loaded.
7) Team YAI: One of the most popular positioning methods for indoor positioning is the fingerprinting method.However, the accuracy of positioning results suffers from the definition of distance among the Received Signal Strength (RSS) values and the fingerprinting table.Because the unit of the received Wi-Fi RSS value is dBm, the similarity calculation is prone to errors when performing fingerprinting positioning.YAI team proposes a fuzzy-based pre-processing method so that the RSS entries in the fingerprinting database can be converted into the corresponding defuzzification values.In the following paragraphs the system is introduced in detail.
a) Fuzzy-based pre-processing: The membership function used is the bell-shape membership function, which could be expressed as follows: where the parameters a, b, and c would affect the width and slope associated with the bell-shape.Using the training data, the parameters in (5) are obtained.Fig. 11 illustrates the membership functions used in this competition.
The defuzzification method used is based on the weighted average formula as follows:

TABLE IV AN EXEMPLARY FINGERPRINT MAP WITH THE PROPOSED FUZZY-BASED PRE-PROCESSING OF YAI TEAM SYSTEM
where x is the received RSS values, μ i (x) denotes the degree of membership for the i -th rule, y i is the weight of the i -th rule, and N = 5 is the number of rules used.Fig. 12 shows the mapping for the defuzzification process used.Using the mapping function as shown in Fig. 12, a fuzzy-processed fingerprint map is obtained.An example to explain the main idea is illustrated in Table IV.Note that "na" denotes the RSS value and it is below the receiving sensitivity of a Wi-Fi module on a smartphone.In this case, the demapping function maps "na" to 0, which means the RSS value is very weak.Then this fuzzy-processed fingerprint map can be leveraged to get the location with the conventional localization method.
b) Simulation results: To evaluate the performance of the proposed positioning algorithm the competition files were used.Using the testing data, the resulting Cumulative Distribution Function (CDF) curves (see Fig. 13) of the errors in positioning are obtained.It can be seen that when locating using only Wi-Fi, the third quartile of the positioning errors are 3.539 m.After improving the fuzzy-based pre-processing, the value could further drop to 2.189 m, thus significantly reducing the positioning errors.
8) Team Indora: The positioning method of Indora team system is designed for the smartphone users in the scenario with known map floor plans, smartphones equipped with sensors, and with no additional infrastructure installed in the building, which is in accord with the competition Track rules.The The proposed positioning system [33] consists of multiple components merged together using the Bayesian filtering.A floor transition is detected using barometer measurements.However, these transitions were also identified using the Wi-Fi fingerprinting method as the incorrect floor detection has significant effect on the overall performance in the competition.The Bayes filter calculation corresponds to the movement on a single floor.The filter probabilistically estimates a current state which is defined as the 2D position on the selected floor.A calculation (consisting of the transition phase and the evaluation phase) is triggered when a step is detected.The transition phase is performed using PDR, i.e., the step direction is obtained from inertial sensors and together with the expected step length introduces a new estimation calculated from the prior position.Noisy sensor measurements, incorrect step length model, and other aspects are resolved by the filter which increases the uncertainty of the current estimation.The uncertainty is reduced during the evaluation phase utilizing the map.The map model (based on the tessellation) is generated from annotated floor plans using a custom tool.This framework was applied on the IPIN 2018 [1] and IPIN 2019 [2] competitions using the centroid grid filter as the Bayesian filtering implementation.Another method for the positioning was derived from the existing system by replacing the grid filter with the particle filter, improved step length calculation, and using Wi-Fi fingerprinting as an additional method for the evaluation phase of the filtering.The position is chosen among particles (or grid cells in the former approach) with the maximum belief value, i.e., the position within the building with the highest assigned probability.
b) Competition strategy: The proposed framework is designed for the real-time positioning.This aspect was taken into account during the off-site competition.Every trial was simulated on the smartphone in the same manner as for the on-site localization.However, the input sensor file was split for the processing convenience to individual files with consecutive steps on a single floor, i.e., the floor transition detection method was performed on raw sensor measurements and all parts were processed separately.
First attempt was computed using the particle filter and the Wi-Fi fingerprinting method.Second attempt consists of the method output using the centroid grid filter.The main focus was on the parameter configuration in this submission.The centroid grid filter was applied on IPIN 2018 [1] and IPIN 2019 [2] competitions.In [33], the method was compared with other grid-based approaches and the recommended configuration was discussed.Multiple parameter settings were explored based on the previous observations leading to the selection of the final competition submission.Third result was obtained as a combination of both approaches.The particular method was chosen for each floor individually according to the author's consideration based on visualized trajectories.
c) Results analysis: The third attempt was a combination of both applied methods.The centroid grid filter was selected on a segment corresponding to the second floor and the particle filter approach was chosen for all other floors.However, the achieved result did not outperform other attempts.Official third quartiles of errors are 6.85 m (particle filter), 8.39 m (grid filter), and 7.02 m (mixed).The analysis with known ground truth positions revealed the incorrect time shift of the first segment on the third floor.Corrected positions obtained the third quartile of errors 3.86 m, 4.46 m, and 3.89 m.For better understanding of the system performance, the first approach was replayed without the Wi-Fi fingerprinting method.The method with the particle filter component resulted in 4.38 m.This result is similar to the centroid grid filter (4.49m).These two systems differs only in the applied Bayesian filtering implementation.The results supported former observations as the grid approach is more stable in the prediction (2.7 m mean of errors) and the particle filter provides a lightweight approach for the computation but it requires additional approach to reduce outliers (4.3 m mean of errors).The Wi-Fi fingerprinting erased large errors on some checkpoints, e.g., seven consecutive positions with errors above 14 m including three positions above 24 m were corrected with the Wi-Fi to reasonable values with the maximum 7.2 m and minimum under 1 m.9) Team TJU: Fig. 14 shows the block diagram of the approach proposed by TJU team.It consists of four stages: 1) the PDR, 2) magnetic fingerprinting, 3) the floor recognition and 4) the trajectory fusion.The proposed approach uses the magnetic fingerprinting and PDR to separately generate the trajectory of the device, and fuses the two estimated trajectories to produce precise trajectory by Kalman Filter (KF).In addition, the current floor is detected relying on Wi-Fi database for each floor, and the transitions between floors depends on air pressure measurements.The details of the four stages are described in the following paragraphs.
a) PDR backbone: The PDR backbone is implemented following two steps: • step detection using peak detection method; • adaptive position updating strategies according to peak values: where r refers to 2D position; p is the timestamp of a specific force norm peak; SL refers to a constant step length; ψ refers to estimated heading; f p refers to the peak value of specific force norm; and γ is the threshold to recognize still state and motion.
Generally, the orientation of the device is estimated using accelerometer and gyroscope readings.The magnetometer was rarely used indoors due to the distortion of the magnetic field.In spite of the drawbacks of indoor magnetic data, filtered magnetic field can still enhance the orientation estimation [34].Inspired by TJU team previous work [35], the original Madgwick algorithm [36] is improved to achieve high-accuracy and robust heading estimation.Fig. 15 shows the flowchart of the proposed heading estimation algorithm.When the agent is still, its attitude is frozen to avoid introducing errors.Otherwise, MadgwickAHRS algorithms based on different data streams are triggered according to quality of magnetic fields.The improved method is immune to magnetic disturbance but long-term motion in magnetically disturbed environment may degrade the performance of the improved heading method.
b) Magnetic fingerprinting: Magnetic fingerprinting can be formulated as a classification problem.Namely, a classification model can be trained using magnetic features labeled by reference position indexes.Instead of using magnetometer readings m b as an observation directly, they are transformed into magnetic vector m n = {X, Y, Z } under navigation frame (n-frame).Since X and Y are changed with directions, horizontal intensity, H = √ X 2 + Y 2 , vertical intensity, Z , and total intensity, F, are used as features to train a machine learning-based model.The benefit is that H , Z and F are immune to sensor orientations and there is no need to collect the data of all directions at a reference point.
During data collection, a participant stays at a reference point for a short while or walks slowly through a reference point.Therefore, the data in 0.5 s before and after the reference point timestamp is considered as that at the point, and extract all data for entire training data set.As a consequence, a raw data set labeled by reference point indexes is built.To enhance the data set, a one-element sliding window with a size of N is used to extract the data of each raw 1 s fragment for each reference point.Then, the extracted data in the sample window is transformed into the feature Finally, several machine learning-based models are trained, such as KNN, SVM, Naive Bayes and ensemble models.After 10-fold cross validation, it can be found that the 1-NN model achieves the highest F1-score of 0.9949.c) Floor recognition: In this stage, the training data are used to create a radio map for each floor using received signal strength (RSS) measurements.RSS values in existing MAC address list are used to train a random forest model to determine the floor ID.However, since the Wi-Fi data is collected with a very low frequency of approximately 0.25 Hz, it cannot provide enough time resolution to determine the precise transitions between floors.Therefore, barometer data with a higher frequency of 5 Hz is used to detect the floor transitions.A mean filter was used to smooth the barometer data before calculating the data difference in successive timestamps.Then, the start and end of the transition between floors can be clearly identified.Wi-Fi data assisted with barometer data can estimate the vertical trajectory of a user well.
d) Fusion using Kalman filter: The PDR system outputs high-frequency 2D positions, while reference points with a lower frequency are recognized by magnetic fingerprinting model.A Kalman filter acts as a bridge to relate two systems.The filter outputs the corrected path using the relative estimations from the PDR model (time update) and absolute position estimations from the magnetic positioning system (measurement update).

10) Team
Next-Newbie Reckoners (NNReckoners): NNReckoners team's method focused on two main parts: position prediction using Random Forest prediction model and IMU position estimate using Wi-Fi propagation and PF (Fig. 16).In other words, the team approached the competition data set with 2 challenges in mind: 1) increasing the volume of Wi-Fi data to train the prediction model, and 2) leverage the IMU to improve position estimate by Wi-Fi.
In order to build the RSS data set to train the prediction model, enough data is required.Hence, data augmentation was adopted to improve the model performance.With the data extrapolation, modifying the RSS at detected AP by increasing the signal strength by 5, the original Wi-Fi data set was enlarged by 30 times [37].
Complementing the Wi-Fi prediction, PDR was applied by incorporating the step count, stride length and orientation calculated from the IMU sensor.The idea of Wi-Fi propagation was to trust the Wi-Fi prediction and recursively estimate the position using the step counts and bearings obtained in between.
Additionally, a particle filter was used to identify the best possible route taken with respect to the derived PDR algorithm.A set of particles was first distributed within the bounds of the map as provided by the competition organisers.Subsequently, a simulation is started from the starting timestamp to the ending timestamp where each particle was moved in accordance to the step data and bearing obtained from the PDR algorithm.Each particle contained a weight and a history of coordinates.If a Wi-Fi position estimate is available, the weight of particles within a radius from the estimated position and its estimated distance error is further increased.It was set due to the fact that Wi-Fi prediction was observed to produce shorter distance errors than PDR.At the end of the simulation, the particles with the highest weight are selected, and the final route is determined by looping through and averaging the history of coordinates for all selected particles.

V. TRACK 4: FOOT-MOUNTED IMU-BASED POSITIONING A. Track Description
Track 4 was dedicated to foot-mounted inertial and GNSS navigation in an off-site context.Data were collected with the PEdestrian Reference SYstem (PERSY) sensor (see Table VI) developed by the GEOLOC team at University Gustave Eiffel.Track chairs collected the data by walking through the competition area over a 1.2 km walk path spanning four different floors, using lifts, escalators and travelators.Also a few outdoor parts were included as shown in Fig. 17, as well as some breaks of various duration.Track 4 followed the same data collection strategies as the off-site competitions organised in previous years [1], [2].In contrast with all the other Tracks, where competitors were provided with a detailed map beforehand and could make use of that information, competitors in Track 4 could not use of any map information.
Two data sets were given to competitors.Data set num 1 was taken on a single static location for several hours, and was meant to be used for sensor calibration, by enabling competitors to compute noise and measurement bias of inertial sensors (Allan variance).Data set num 2 was the data recorded on the Atlantis shopping mall area following 6 different steps, as shown in Table V and in Fig. 18.The competitors' objective was to re-build the trajectory realised by the Track chairs.The evaluation was done by comparing 2D position and floor level estimated by each team to the coordinates of 67 reference points (key points).To do so, a Table containing timestamps of expected key points was shared, and competitors had to provide the corresponding coordinates.

TABLE VI INFORMATION ABOUT EMBEDDED SENSORS INSIDE PERSY
Data Set and supplementary materials -e.g.data sheet of sensors embedded in PERSY-were provided to competitors of Track 4. These contents and the ground truth location for evaluation are now available for further benchmarking in [38].This package complements the ones from the previous editions [39] and [40].

B. Competition Area
For IPIN 2020, due to Covid-19 situation, Track4 competition was held in "Atlantis Le centre", a large shopping mall close to Nantes -France.This site has already been used in IPIN 2018 for all competition Tracks, and a very accurate survey was realised.This has eased the design of the ground truth (see [1] for details on the survey).There were multiple difficulties when surveying such a big shopping mall: wide areas, lifts, escalators, and even a carousel, as illustrated in Fig. 19.Complexity related to the Covid-19 also led the Track chairs to make loops on the path in order to respect the direction of travel, as shown in Figure 20.

C. Indoor Positioning Solutions Provided by Competitors
1) Team WHUGNSS: The classic zero-velocity update algorithm (Zero-velocity update (ZUPT)) based foot-mounted pedestrian dead reckoning consists of a strap-down inertial navigation algorithm, a stance phase detection algorithm, and an error state Kalman filter.However, the classic ZUPT-based Foot-PDR [41], [42] cannot overcome the influence of the complex motion of the pedestrian.The WHU-GNSS team system is based on several schemes designed to improve navigation performance, as shown in Fig. 21.
The core algorithm is the strap-down inertial navigation algorithm.On this basis, a zero-speed detection method with adaptive threshold setting is used to adapt to different users.Next, the motion pattern recognition algorithm is used to distinguish whether the user is walking normally or taking the escalator and elevator, and uses constant speed, Zero-velocity update (ZUPT), Zero angular rate update (ZARU), Improved heuristic drift elimination (iHDE), linear trajectory and height constraints to improve the position estimation accuracy according to the discrimination results.In addition, the magnetic field will be used to detect whether the user has returned to the place where they have walked, so as to correct the current navigation state with the historical estimated position.And when the user comes to an outdoor scene, the GNSS signal will be used to improve the final positioning performance.
a) The multi-constraint algorithms: The classic Generalized Likelihood Ratio Test (GLRT) method is one of the most common algorithms for detecting the stance phase [43], [44].In the WHU-GNSS team system, an improved adaptive threshold is used instead of the fixed threshold method to detect the stance-phase in each gait cycle.The adaptive threshold method is adaptable to different gait frequencies in dynamic motion.Once the stance phase is detected, a zero velocity vector is used to estimate and correct the navigation error [41].
The heading angle error and the z-axis gyroscope bias of the ZUPT algorithm are unobservable.Thus, the following methods are used to constrain the error divergence of the heading angle.First, the Zero angular rate update (ZARU) algorithm is employed to estimate the gyroscope bias and heading angle error [45].Compared with the stance phase detection algorithm, a stricter fixed threshold is used and a more extended continuous period to determine the update chance of ZARU.Second, when a pedestrian is determined to be walking in a straight-line path or the corridor's primary orientation, the Improved heuristic drift elimination (iHDE) and the straight-line constraint algorithms are applied to estimate the heading angle and the z-axis gyroscope bias [46], [47].These algorithms can effectively improve the performance and reliability of pedestrian navigation.
The height error divergence is also a significant problem in Foot-PDR, especially for multi-floor navigation and positioning applications.In the absence of a barometer, an effective height constraint algorithm is adopted to reduce the error drift along the vertical channel.When pedestrians go up and downstairs, the slope angle can be considered constant in most cases [48].In the WHU-GNSS solution, the stride length and the slope angle between adjacent footsteps are used to determine whether the pedestrian is walking on a plane or going up and downstairs.Then the slope-based or plane-based height constraint algorithm is used to improve the estimated height accuracy in Foot-PDR.
The other extreme scenario is the escalator or lift.Usually, escalators run at a constant speed.When a pedestrian stands relatively static on the escalator, the specific forces measured by the foot-mounted IMU are almost all derived from local gravity.The gravity information can be fused in a tightly coupled manner in the WHU-GNSS solution, so the drifting error can be constrained even when a pedestrian stands still on an escalator.Moreover, when the pedestrian takes a lift, the specific forces (i.e., the accelerations) will exhibit clear acceleration motion and deceleration motion process.The vertical (up or down) velocity information of the pedestrian can be estimated using acceleration and deceleration motions.Thus, the vertical velocity can be as observation information to improve the performance and stability of the Foot-PDR.
Many ferromagnetic materials exist in indoor building structures.So, magnetometers cannot be used to determine the heading angle in Foot-PDR directly.Yet, combined with a rough position, the magnetic field signals can recognize similar areas when the pedestrians return to places they have walked before.This meaningful information can help improve the robustness of Foot-PDR in practical application.
The Foot-PDR is integrated with GNSS signals in a loosely-coupled manner [42].Satellites with small elevations should be discarded to avoid the gross error as much as possible.Besides, some measurements with low quality judged The scheme of foot-mounted PDR system based on multi-constraint algorithms proposed by AIR team.
by the innovation vector's magnitude and covariance need to be rejected in the Kalman Filter (KF).Furthermore, the adaptively robust filtering algorithm is used to control the effects of inaccurate measurements in the WHU-GNSS solution and improve system accuracy.
The optimal inertial sensor parameters (i.e., the bias instability of gyroscopes and accelerometers, the angular random walk, and the velocity random walk) are determined through the provided long-term static data.The magnetometer is also calibrated through the classic ellipsoid fitting method.
2) Team AIR: The pedestrian foot-mounted PDR system proposed by AIR team is shown in Fig. 22.
In the above framework, five constraint algorithms are included in the middle modules: Stance & Still Phase Detection, the Heuristic Drift Elimination (HDE), the Height Update Algorithm (HUPT), the Zero-velocity update (ZUPT), and the Earth Magnetic Yaw.Meanwhile, the Stance & Still Phase Detection includes two components: the Generalized Likelihood Ratio Test (GLRT) detector algorithm used under the condition of the slow and normal pedestrian gait speed, and the Hidden Markov Model (HMM) detector algorithm used under the condition of the dynamic and fast pedestrian gait speed.After that, using the improved HDE and HUPT method to estimate current position errors, ZUPT is used to estimate the velocity error, while Earth Magnetic Yaw based on quasi-Static Magnetic Field (QSMF) method is used to estimate the heading error.
a) The multi-constraint algorithms: A gait or a walk cycle consists of two phases: the swing and stance phase.In the swing phase, the foot is not in contact with the ground.In contrast, the foot contacts the ground in the stance phase.GLRT algorithm has obvious advantages for zero speed detection of stable pedestrian gait velocity, while HMM algorithm has a good effect for zero speed detection of dynamic and fast pedestrian gait speed.Thus, the two methods are combined to achieve the dynamic human stance & still phase detection [49].When the Stance & Still Phase Detection detects the stance and swing phases of human foot gait from the data from IMUs, ZUPT method is used to constraint the velocity divergence [50].
HDE algorithm is a very useful method to constraint the system's heading drift, if the indoor reference heading can be known in advance.In the AIR team method, the initial heading is used to calculate several possible reference directions of pedestrian walking [48].Then, unlike the existing HDE method, which mainly corrects inertia recursive heading, the closest reference direction is used to calculate the estimate position at the current footstep, then uses the position error between the estimate position and the inertia recursive position to restrain the position divergence.The procedure is shown in Fig. 23.
Height divergence is a major problem in Inertial Navigation System (INS)-based foot-mounted PDR system in multi-story positioning.If a pedestrian is walking on a plane, the slope of the current stride is approximately zero degree, if that, keep the height always unchanged.While walking on a staircase, the method proposed uses the actual slope of the stairs (usually 20 45 degrees) to calculate the height change of the current stride, which can be used to constrain the height divergence of the current stride [48].If pedestrian is on an elevator or escalator, it mainly can be effectively determined by analyzing the characteristics of acceleration, especially the acceleration in the vertical direction.
The magnetic field is very useful to estimate the heading of the system, but the magnetic disturbance has a severely effect on the estimation.In AIR team system, an improved QSMF method combined with a compass filter is used to estimate the heading in the perturbed magnetic field [51].In addition, in areas where pedestrians repeatedly walk, a series of magnetic sequence information is used for pedestrian trajectory matching to improve the effect of heading constraint.
3) Team Free-Walking: The positioning system proposed by team Free-Walking is shown in Fig. 24.
The Free-Walking system combines data pre-processing, motion mode recognition, INS mechanization, adaptive zero velocity detection, ZUPT-aided Kalman Filter (KF) and altitude constraint.The data pre-processing includes sensor calibration, filtering and Coordinate system transformation.After pre-processing, motion mode recognition algorithms are used to help adaptive threshold ZUPT detection.Then a ZUPTbased KF is used to get position information.Meanwhile, motion mode results is also used to constraint height error.
a) Error-constraint method based on walking mode: For pedestrian positioning, the human motion modes describe the overall movement of pedestrians.The pedestrian motion modes are particularly important for pedestrian navigation, while the pedestrian motion modes are variable during the procedure of pedestrian navigation.Therefore, a walking mode classifier is designed (see Fig. 25) based on the stacked denoising autoencoder [52] and temporal Convolutional neural network (CNN) with attention to recognize eight pedestrian motion modes [53], [54].
ZUPT-aided INS has ability to suppress navigation errors.Free-Walking team uses the periodic gait-cycle window to divide the pedestrian movement into discrete gait cycles; then, the minimum value in each gait cycle is taken as the zero-speed state point.The time length of the gait cycle is different under different motions.The gait-cycle duration is adaptively adjusted based on the classification result of walking mode to adapt to various pedestrian motions [55].Compared to the existing methods, the proposed method does not need to set the zero-speed detection threshold, and performs well for zero-speed interval detection under various pedestrian movements.The stationary state of the foot during the stance phase is taken and feeds the zero-velocity information (pseudomeasurement) into KF to compensate for the velocity, the position and the attitude errors.
The height errors in Strapdown Inertial Navigation System (SINS) solution will grow without boundary and cannot be eliminated by ZUPT measurements.When a user walks on the same floor, the altitude does not change.The altitude changes only when the user goes up and down stairs.Therefore, the vertical displacement of pedestrian is constrained by two factors: stair height and motion mode.If the height of each stair in a multi-floor building is fixed, the height of each gait cycle is determined by the number of walking stairs in that gait cycle.Therefore, the classification result of walking mode is used to constrain the height error.
4) Team BHSNIP: The Pedestrian Navigation System (PNS) based on Inertial navigation system-extended Kalman filterzero velocity update (IEZ) -also referred as INS-EKF-ZUPTis widely used in complex environments without external infrastructure owing to its characteristics of autonomy and continuity.However, due to the poor observability of heading errors to ZUPT and the instability of vertical inertial channels, further corrections of the estimated trajectories under the IEZ framework are still needed to obtain higher positioning accuracy.
In order to achieve high performance for PNS in terms of accuracy and robustness, BHSNIP team integrates the Micro-Electro-Mechanical Systems-Inertial Measurement Unit (MEMS-IMU) and Global Positioning System (GPS) as shown in Fig. 26.In this scheme, MEMS-IMU provides the 3-axis accelerometer, 3-axis magnetometer, and 3-axis gyroscope readings which are [ f x f y f z ], [mag x mag y mag z ], and [ω x ω y ω z ] in the body frame, respectively.The main work has the following features: 1) Aiming at the weakly observability of heading drift for MEMS-IMU, the iHDE algorithm is proposed.The algorithm has the following three steps: First, heading information is extracted from pedestrian's straight-line motion track, which is used to construct four or eight datum directions of the building; second, building heading information is utilized to estimate yaw errors of trajectories that satisfy specified rules; and third, these yaw errors are utilized as the EKF observation to estimate the state error of the navigation parameters.2) In order to deal with the problem that the inertial vertical channel is unstable under the traditional IEZ framework, which makes it impossible to locate the floor by SINS solutions, the improved step height equidistant (ISHE) is exploited.At the beginning, the adaptive network-based fuzzy inference system (ANFIS) is used to identify different vertical modes including elevator, escalator and staircase (walking upstairs, horizontal movement, and walking downstairs).Then, the floor information or altitude is estimated by ISHE.3) To detect the stance phase accurately, adaptive-ZUPT algorithm is used based on backward neural network.
In conventional researches, positioning performance is easily affected by the ZUPT with fixed threshold, because it is difficult to determine ZUPT conditions for jump, fast walking, running.

4) GPS is fused with MEMS-IMUMEMS-IMU through
Robust Extended Kalman Filter (REKF), which can remove the contaminated points of GPS signal.What is more, GPS can provide global coordinates.Fig. 27 shows the horizontal trajectory.The estimated track starts from the red circle and the blue line represents the moving trail of the pedestrian based on the proposed method.The positive direction of abscissa and longitudinal represents   East and north respectively.The track in the figure is shown in relative coordinates that will be transformed into the WGS84 coordinate system.Fig. 28 illustrates the three-dimensional trajectory.The estimated track also starts from the red circle and the blue line represents the moving trail of the pedestrian based on the proposed method.The x-axis, y-axis and z-axis of the coordinate system represent east, north and up respectively.The relative coordinates representing the track in Fig. 28 will be transformed to the WGS84 coordinate system.

VI. TRACK 5: XDR CHALLENGE IN MANUFACTURING 2020 A. Track Description
The purpose of Track 5 is to evaluate the practical performance of indoor localisation methods under realistic industrial scenarios.Indoor localisation competitions have been held, named "PDR Challenge" or "xDR Challenge" as the official competitions or the relevant event in past IPIN conferences.Track 5 is a sequel of the PDR/xDR Challenge series competition, which is named as "xDR Challenge in Manufacturing 2020".In this year's competition, the competitors are asked to estimate the trajectory of employees working in the factory and forklifts driven in the factory.
As specific industrial scenarios, the target for PDR Challenge 2017 and xDR Challenge 2018 were picking operation in a warehouse [56], while for xDR Challenge 2019 it was serving in a restaurant and manufacturing operations in a factory.The scenario of the competition for 2020 was manufacturing operations in a factory.Competitors were required to estimate operators' trajectory and forklifts' trajectory in the factory by utilising indoor localisation methods based on dead reckoning algorithm, positional correction methods with Bluetooth Low Energy (BLE) beacons and other information provided.
Characteristics of Track 5 can be summarised as follows: 1) Utilising the Data Actually Used in the Operation: Similar to other Tracks, Track 5 aims to compare practical performance of indoor localisation methods or systems under realistic industrial scenarios.Its most remarkable characteristic is that the data provided to competitors is obtained from by an analysing system for manufacturing operation which was used during real operation [57], after approval of provision of the data actually used.
The operators are carrying Android devices which measure sensor data for the analysis based on indoor localisation.This means that in Track 5 data is not provided by an actor following a predetermined path, but by real operators doing their daily job.This adds significant difficulty in estimating the trajectory with respect to other Tracks, mostly because the target movements include various types of motion during the manufacturing operations, rather than simply walking at constant speed and staying still for a while.
As the data set, we provided measured sensor data that include angular velocity, acceleration, magnetism, atmospheric pressure, and RSSI of BLE beacons.Also, partial ground truth positions are provided for correcting the position.These ground truth data are assumed to be available from the record of the operations and required for long-term estimation by indoor localisation.The lengths of the data are in units of working hours.The lengths per data are about 2 hours to 7 hours.

2) Evaluating Dead Reckoning Methods for Various Types of
Moving Objects: The PDR/xDR Challenge series competitions deal with indoor localisation methods based on various types of the dead reckoning methods.Dead reckoning for vehicle is called Vehicle Dead-Reckoning (VDR).The term "xDR" is used to indicate various types of dead reckoning.The target of the Track 5 is not only operators working in the factory, but also forklifts driven in the factory.Dead reckoning of the vehicle such as the forklifts is a quite challenging topic.Thus, there are two separated sub-Tracks for PDR and VDR.
3) Multi-Faceted Evaluation of Performance for Indoor Localisation Methods: In order to evaluate practical performance under industrial scenarios, multi-faceted evaluation metrics has been used.The evaluation metrics in the PDR/xDR Challenges has been revised.As the evaluation metrics for this year's competition, a three-evaluation indicators and three-negative check criteria were adopted as follows: Evaluation indicators about error • Absolute error -Circular Error (CE): absolute 2D positional error compared with ground truth position.• Error distribution bias -Circular Accuracy (CA): evaluating degree of bias of error distribution in 2D error space.
• Error accumulation gradient (EAG): evaluating speed of error accumulation caused by relative tracking with dead reckoning.

Negative Checks
• Requirement of moving velocity: checking if local moving speeds in the trajectory are less than a defined threshold.
• Requirement of validity of trajectory: checking the incursion of the trajectory into un-walkable area.• Coverage ratio: check if each evaluation point has corresponding submitted results.Each evaluation indicator and criterion are converted into evaluation indexes up to 100 and weighted summed for calculating the integrated index which determines the winner of the competition.We adopted median of CEs (C E50) as an indicator of the absolute error.The error accumulation is the one of main concerns in relative tracking method such as xDR.In order to evaluate the error accumulation, BLE signals in the data set have been intentionally and partially deleted [56].Partial ground-truth position is provided for error correction and for evaluating the speed of error accumulation from the correction points where the ground truth position is provided.Competitors are required to deal with these unique characteristics of the data set.C E75 has not been used for determining the winner, but only for comparison according to the EvAAL framework.However, C E75 can be easily calculated by using our evaluation script for calculating evaluation indicators and negative checks.Please refer to the script shared on the GitHub for further details [58].

B. Competition Area
The target field for the PDR subtrack is shown in Fig. 29.The target field of the VDR subtrack is shown in Fig. 30.We provided some examples as sample data sets.In the figures, examples of the movements of an operator and a forklift are shown in blue dots.The yellow dots represent examples of the partial ground truth data for correcting the positional errors.The black coloured areas represents the un-walkable areas.Competitors are able to avoid the incursion into the un-walkable area by using map matching techniques.BLE beacons are arranged in the target area for absolute localisation and positional correction.According to the demands of the factory for maintenance, solar-powered BLE beacons, Fujitsu's PulsarGum, are used.The interval of signal emission is 1.26 s at minimum, but it is not guaranteed and varies in proportion to the amount of generated electricity.Competitors are required to deal with this characteristic of the beacon.

C. Indoor Positioning Solutions Provided by Competitors
1) Team KawaguchiLab: Team KawaguchiLab has studied IMU-based indoor localization using smartphone.In the 2020 competition, the challenge was to integrate Kawaguchi-Lab IMU-based research with non-IMU sensor based system (BLE, map information), and to build a robust indoor positioning system.KawaguchiLab system is simple because it makes no complex assumptions.Therefore, even in a Track 5 environment where there are few movement constraints, it works   robustly, although there is a trade-off for some loss of accuracy.Fig. 31 shows overview of KawaguchiLab team system.It consists of three phases: denoising, dead reckoning, and compensation.
a) Denoising phase: Gyroscopes have an offset that depends on the inherent characteristics of the sensor and temperature.It causes a serious cumulative errors in dead reckoning.Hence, they are removed using real-time offset removal algorithm: First, whether the sensor is stationary or moving is obtained with an Fast Fourier Transform (FFT) based method; second, the offset by averaging the angular velocity while stationary is calculated.Finally, the angular velocityis calibrated using the latest updated offset.
b) Dead reckoning phase: In speed estimation, Deep Neural Network (DNN) base method is used [59]- [61].Deep neural network architecture consist of Long Short-Term Memory (LSTM) and full-connected layer LSTM extract time series features of 3-axis acceleration by sliding window and full-connected layer converts time series features to speed.This approach gains robustness to noisy data and work with various gaits.
In heading estimation, first, gravity direction ĝDC S is estimated using Multiplicative Extended Kalman Filter (MEKF) [62].DCS represents the device coordinate system.Secon, angular velocity ω DC S is projected to gravity to get the horizontal angular velocity ωGC S z .GCS represents the global coordinate system.Projection process is as follows: Finally, the heading is calculated by integrating time-series horizontal angular velocity.Integration process is as follows: c) Compensation phase: Pseudo reference position from BLE signal is generated to compensate trajectory.BLE signals are searched using sliding window for about 10 s.Then, the distance from BLE beacon to subject is estimated using three or more BLE signals.A pseudo reference position by using these distance.
Similarity transformation model [63] is used to compensate the trajectory using the true reference position and pseudo reference position.The parameter of this model is updated using similitude ratio.The similitude ratio s is calculated by using actual moving distance d and estimated moving distance d e .
Finally, the α is multipied to the estimated position change.
The path is generated using map image as physical constraints to avoid obstacles.The shortest path from one reference point to the next one is calculated and with astar algorithm the next reference point is searched.
2) Team YONAYONA: YONAYONA team indoor positioning technology is implemented in two stages: absolute position determination using BLE signals and map matching using map information.Using the acceleration and angular velocity measured by the IMU is a relative positioning approach, which often causes drifting errors.Therefore, YONAYONA system first efficiently estimates the location based on the RSSI, position, and signal strength parameters, and then corrects for the natural behavior of the person's walking speed and direction.A major challenge for this algorithm is to deal with the situation when the number of observed BLE beacons is not enough or when there is a wall between the previous predicted position and the next predicted position.
a) Absolute position determination: Since the number of BLE beacons observed is not constant, absolute positioning is calculated by selecting three beacons with high RSSI at a certain time (every 0.5 s in this implementation).The distance between the observer and the beacons can be computed by RSSI and Ptx (measured RSSI 0.1 m away from the beacon).As a result of trying various approaches to estimate the position based on the distance data, such as trilateration, position averaging, and position averaging with power value weighting, the average-weighted method, which has the least error, is applied in this algorithm.
b) Map matching: In this section, an algorithm is build to predict realistic human movement based on the map data provided by the competition organizers.In cases where a line connecting two points estimated by absolute surveying would encroach into a wall, an inaccessible point with a nearby accessible one is replaced.Then, the estimated points are connected with each other in a smooth trajectory so that the walking speed can be kept within a sensible range.
c) Problem: This algorithm relies on absolute position estimations, which makes it difficult to deal with situations where there is a large error in the value of the signal received from the beacons, or where the number of signals received is not sufficient.In this implementation, the estimation accuracy within the Absolute Localization Inapplicable Period (ALIP) time set at a specific time was reduced, resulting in a larger error.In fact, there were not enough time to build an algorithm that also implemented PDR and VDR by the competition deadline, so it is not possible to refer to relative positioning.A possible improvement to this technique is to design a robust system using the Kalman Filter (KF) from two estimates, one for absolute positioning and one for relative positioning.

VII. TRACK 6: SMARTPHONE-BASED VEHICLE POSITIONING WITHOUT ADDITIONAL EQUIPMENT A. Track Description
The goal of Track 6 is to evaluate the performance of different integrated navigation solutions based on the sensors of vehicle-mounted smartphone, such as GNSS, MEMS and magnetometer, etc.A Huawei mate20 smartphone was used to record raw multi-sensor data in the vehicle scene and a reference system based on Differential Global Navigation Satellite System (DGNSS) and Fiber Optic Gyro Inertial Navigation System (FOG-INS) with an expected accuracy of 5 cm at 1 Hz provided the ground truth.Two data sets were provided.The first one containing the ground-truth reference was used for sensor and algorithm calibration.The second one was for the calculation of the coordinates and accuracy evaluation.

B. Competition Area
The test route of Track 6 (see Fig. 32) includes an outdoor scenario with unobstructed satellite view, an attenuation scenario with partially obstructed view and an indoor scenario without satellite view.In the test process (see Fig. 33), there were several long interruptions of GNSS signal and an irregular test route was adopted.Besides the navigation measurements derived from the sensors installed in smartphone, there C. Indoor Positioning Solutions Provided by Competitors 1) Team WHU&AutoNavi: WHU&Autonavi Team system uses GNSS/INS integrated positioning as the basic algorithm and focus on making full use of vehicle motion constraint information and magnetometer observations to provide stable positioning services.Fig. 34 shows the flowchart of the vehicle integrated positioning algorithm based on smartphone built-in sensors.And the algorithm can be divided into 3 parts: 1) GNSS/INS integrated positioning algorithm (the red dotted part), 2) the vehicle motion model constraints (the orange part), and 3) magnetic heading constraint (the green part).
a) GNSS/INS integrated positioning algorithm: GNSS/ INSintegrated positioning is the most basic and backbone algorithm in vehicle positioning scenarios.INS is used as a bridge to correlate all available observations, and GNSS, as the only available absolute positioning method in the offline mode of the smartphone, determines the positioning performance of the system.
INS mechanization is employed to integrate the gyros and accelerometer output.Due to the low performance of the  smartphone built-in sensors, the influence of the angular rate and sculling effect caused by the rotation of Earth and motion speed can be ignore [64], [65].Therefore, the rigorous INS mechanization can be simplified to achieve more efficient calculations.
An Extended Kalman Filter (EKF) is employed to fuse GNSS and INS for reducing the error caused by non-linearity.And the 20-dimensional navigation error state includes position, velocity, attitude, gyroscope bias, accelerometer bias, misalignment angle (the angular difference between the smartphone built-in sensor and the vehicle coordinate system), and the lever arm parameters (the offset of the sensor measurement center to the center of the vehicle coordinate system).To maximize the navigation performance of the sensor, the performance parameters of the gyroscope and accelerometer are adjusted according to the three sets of training data given by the competition.
For smartphones, the distance between the GNSS antenna and the IMU measurement center is very close (e.g., several centimeters), and the GNSS position accuracy in single-point positioning mode is at the meter level, so the GNSS antenna and the IMU measurement center can be considered to overlap.Besides, since the standard deviation cannot accurately determine the true positioning accuracy of the GNSS position, the chi-square test is used to eliminate the gross errors in the GNSS position to ensure the reliability of the filtering [66].

b) Vehicle motion model constraints:
To deal with scenarios where GNSS signals are interfered in a complex environment, the vehicle motion constraint model is fully used to improve the relative positioning capability of the system.WHU&Autonavi ystem simply divides the vehicle motion state into stationary and moving by using the raw output of gyroscope and accelerometer.
Stationary state: When the vehicle is judged to be stationary, it can be considered that the speed of the vehicle is zero, that is, Zero-velocity update (ZUPT).ZUPT is an effective means to control the accumulation of velocity error.At the same time, the heading of the vehicle should remain unchanged, and all heading errors can be considered to be caused by sensor errors.The WHU&Autonavi system stores the heading angle at the initial moment of the stationary period and constructs a virtual heading angle observation value, so as to achieve the purpose of effectively controlling the accumulation of heading angle error, called Zero Integrated Heading Rate (ZIHR) [67].
Motion state: For the normal driving behavior of ordinary users, the vehicle will only move forward or backward.Based on such objective facts, it can be assumed that the lateral and vertical speeds in the vehicle coordinate system (that is, the v system) are always zero [67].However, the forward speed of the vehicle still cannot be accurately obtained.WHU&Autonavi Team system uses rticl supervised learning method to train the vehicle forward speed prediction model [68], and the error can be controlled within 0.5 m s −1 .
Due to the random disassembly and reinstallation of the smartphone, the problem of the installation angle and lever arm parameters is not fixed.At this time, traditional direct setting or pre-calibration methods do not have the conditions for implementation.Automatic calibration of the installation angle and lever arm parameters can make the vehicle motion constraint algorithm more applicable.
c) Magnetic heading constraints: The magnetic interference caused by the vehicle shell can be equivalent to the magnetometer bias.So, the heading angle calculated based on the magnetometer observations can still accurately reflect the true heading angle change after the calibration and deduction of the magnetometer bias.Besides, the quasi-Static Magnetic Field (QSMF) is employed for avoiding environmental magnetic interference [69].
2) SZU-Mellivora Capensis: The data collection of Track 6 is located near the Beijing Haidian Airport.Its goal is to evaluate the performance of vehicle navigation solutions based on the integration of different sensors such as GNSS, MEMS, and magnetometers on in-vehicle smartphones.This test is under typical urban road conditions.The smartphone is fixed inside the vehicle, and data is collected through the phone sensor.A single test process lasts about 1 hour and the test route consists of static initial alignment phase (about 5 minutes), open environment phase (about 20 minutes), obstructed environment phase where the GNSS signal is attenuated or blocked by the surrounding buildings or trees (about 25 minutes, during which the GNSS positioning results will be frequently interrupted) and no GNSS signal phase (underground parking lots about 10 minutes, with no GNSS positioning results).The driving process of the test vehicle includes going straight, left/right turning, reversing and parking.
To get the update of the vehicle's position, SZU-Mellivora Capensis team system obtain its velocity and heading.As for the velocity update, the system uses accelerometer and gyroscope, extract their data and align the coordinates, and then train them through proposed Deep Neural Network (DNN) to get the predicted velocity.The same is true for the heading prediction, but raw data used comes from the gyroscope, the magnetometer and the AHRS.Based on the prediction of velocity and heading, the relative displacement of the vehicle can be inferred.Then, the federated filter is used for data fusion.The weight factor is modified through observability to improve the filter and achieve high-precision localization.Finally, a smoothing filter is applied in this method.
The traditional inertial dead-reckoning mentioned above to estimate the motion of the vehicle is a challenging problem.To reduce this unavoidable inertial drift, a data-driven approach is used to inertial tracking.Referring to the network structure on IONet, the motion state of the vehicle is predicted by a trained deep Recurrent Neural Network (RNN).The RNN maintains the local hidden state within a time window, and then extracts the potential features of the time series.These features affect the state output at the next moment, thus enabling an effective recovery of the potential connection between data features and vehicle motion.The time window size is chosen as 1 s (50 frames).The data within the window are (n × 3 × 50) dimensional long-term dependent feature vectors constructed by stacking aligned n sensors.The changes of _v and _h in 1 s can be predicted by Equation 12: Unlike previous data-driven-inertial tracking work, the regression of the displacement vector is split into two separate parts: velocity estimation and heading estimation.The division of the regression task reduces the impact of extraneous sensors on prediction accuracy.In the velocity estimation part, input data are the 3-axis accelerometer and 3-axis gravity sensor data for a one-second period, which are corrected for the coordinate system alignment described above.The output is the average velocity over this time period, based in the two-dimensional plane.In the heading estimation section, input data are the 3-axis gyroscope and 3-axis magnetometer data during the time period, and the output is the sum of the heading changes in one second.The above input data is the best combination of sensors after the experiments performed.
Fig. 35 shows the RNN framework proposed in this system.A two-layer Long Short-Term Memory (LSTM) is used as the core module to solve the gradient explosion and vanishing problem of traditional RNNs, and it can effectively exploit the long-term dependence of time series.Each LSTM layer has 256 hidden nodes well above the dimensionality of the input data.This is in order to give enough inputs to the LSTM units so that the LSTM can fully utilize its function of selecting useful information.To avoid the overfitting problem, a dropout layer is added after each LSTM layer to increase the orthogonality between the features in each layer.Finally, a fully connected layer is placed to regress the velocity and heading changes, respectively.The loss function is defined in terms of the mean square error between the motion parameters and the ground truth.The ADAM optimizer is chosen to minimize this loss value and learn to obtain the best parameters within the RNN.
After obtaining the velocity and heading, the trajectory points can be expressed as: 3) Team YAI: YAI Team system uses three types of sensor data in this competition, namely ACCE, AHRS, GNSS.In the data pre-processing part, the ACCE and AHRS data were averaged per second to obtain data with a frequency of 1 Hz, while the missing GNSS were marked.First, the displacement of the vehicle per second is obtained by adding the initial velocity of the original GNSS to the ACCE data.Then, the YAW angle data of AHRS is initialized.After setting the initial direction angle, the angle ranges from minus 180 to 180 degrees.
The proposed framework used Kalman Filter (KF) for tracking.Fig. 36 shows the flow chart of the proposed tracking framework.The prepossessed data was introduced into the Kalman filter and GNSS to get KF gain to correct the error.Because the KF relied on the previous path to calculate, it does not work well during the missing section and may  produce significant cumulative errors, especially when the device encounters a large-scale angle variation.Fig. 37 shows the estimated driving direction during the testing period.The angle variation is obtained by the difference among a short period, 5 s in this competition.The framework calculated the missing sections of each route depending on the degrees of the angle variation with a threshold of 90 degree.For the small-scale angle variation, the above method was used to compensate for the missing section.For the large-scale angle variation, the KF was changed to one-dimensional to address this issue.To be more specific, KF gain is let to not change the lateral displacement, while only affecting one-dimensional displacement.Afterwards, the AHRS is directly multiplied instead to estimate the direction of travel.

VIII. TRACK 7: CHANNEL IMPULSE RESPONSES A. Track Description
Environments with complex Radio-Frequency (RF) propagation conditions such as indoor, urban or industrial environments have been a challenge for the RF positioning community for a long time.Especially in industrial environments, the abundance of metal objects causing absorption, reflection, diffraction and scattering of the signals leads to highly complex signal propagation that is hard to model analytically.Therefore, classic RF positioning methods relying on multi-angulation or multi-lateration are difficult to apply.
Received Signal Strength (RSS) based positioning exploiting the spatial significance of the propagation conditions has been used in these environments for many years.However, recently the use of Channel Impulse Responsess (CIRs), containing information on the whole signal propagation path, including Multipath Component (MPC) has been proposed.While the specialised hardware and firmware components used to obtain these signals are not yet available in mass user products like smartphones, the introduction of Ultra-Wide Band (UWB) technology into newer generations of devices means that CIR-based positioning is a promising possibility even for low-cost applications in the near future.Since CIRs contains a variety of spatial and environment-related information, it has been used for positioning in three different ways: • Model error mitigation [70], [71]: CIRs are used to classify propagation conditions like a missing line-ofsight (LoS) link or to estimate model errors caused by multipath components.The goal is to use CIRs to enhance classic positioning methods.
• Fingerprinting [72], [73]: CIRs is assumed to be spatially significant and the relation of the signal propagation to the environment is exploited for positioning by implicit modelling using a set of pre-recorded training data.• Multipath-SLAM [74]- [77]: CIRs is used to jointly estimate the position of virtual anchors (i.e.characteristic reflection points) or other significant features and the user positions.Since, in recent years, many research groups have been working on CIR-based positioning in adverse environments, a data set to compare different approaches under a common evaluation framework is highly beneficial to the community.Hence, a robotic scenario data set has been introduced using the popular Decawave DW1000 UWB chip.For this, an industrial environment in a testing hall has been reconstructed, equipped with state-of-the-art positioning reference systems.

B. Competition Area
The environment resembles an industrial setting: it includes metal shelves, industrial vehicles and other objects that   2) and ( 4) and a large metal box (5).The anchor node positions are depicted as red squares.
influence the radio signal propagation in the environment.The measurement setup is depicted in Fig. 38: the stationary anchor nodes (highlighted with red boxes) are placed around the area and the mobile node (highlighted with a green circle) is attached to a wooden table to ensure a constant height.The measurement setup was such that the mobile node was configured as a transmitter and the stationary anchor nodes were configured as receivers.The wooden table was moved throughout the environment at constant speed (as best as possible).Hence, the environment and data resembles a robotic scenario.
Fig. 39 shows the distribution of the acquired data within the environment.The trajectories of the transmitter nodes are in-between the various objects; the size of the area is approximately 1 m × 20 m.In total, about 300,000 channel impulse responses were captured over a time period of approximately 1.5 h.The sampling interval of the data was about 10 Hz.For clearance, a constant sampling interval is not available, as straightforward re-sampling of CIR data is not possible because of the complexity of the signals.Of these, a temporally coherent set of 230,000 CIRs is available for training purposes, while another coherent set of 70,000 data points is used for testing/evaluation.This corresponds to exactly 20 minutes of recording time.A detailed description of the file format and the system specifications as well as downloadable links for the anchor/node configuration and the training and test data is available at http://evaal.aaloa.org/images/2020/ta7-v4.pdf.
C. Indoor Positioning Solutions Provided by Competitors 1) Team YAI: First, the proposed system by team YAI converts the real and imaginary parts of the Channel Impulse Responses (CIR) into the magnitude, where the phase information is removed [78], [79].All CIRs collected were organised by each receiver at the same location and at the same time.
After processing the CIR signals, the proposed system utilizes deep learning to build a mapping between the CIR magnitude and location.In the indoor environment, the size of 1 m 2 is taken as a grid, and then the position where the existing data appears is divided into 15 × 19 grid cells, numbered 0-284.Then, each grid is regarded as a class and each classifier is trained for a reviewer.This way, the positioning problem can be viewed as a classification problem.A typical machine learning DNN follows, to train a classifier for a receiver.In the learning procedure, the temporal magnitude CIR is directly regarded as a static feature vector to learn the grid information by DNN.The DNN network architecture used in the experiments include 8 layers, 465,373 neurons, and the activation function is softmax.A categorical-crossentropy follows to train the network parameters.Fig. 40 shows the network architecture of the used DNN.Finally, six receivers contain six independent classifiers that are able to convert the CIR magnitude into a grid.
Finally, the proposed system uses an ensemble approach to combine the estimation results from the six classifiers.In order to make the final answer more precise, the voting method is used first.That is, the final result is obtained by the majority voting from the six receivers.If voting from the six receivers results in a tie, the proposed system will select the receiver that contains the strongest CIR magnitude.This is under the assumption that the strongest CIR magnitude encounters less deviation and interference.The answer reported by that strongest receiver is viewed as the final estimation.If all six receivers have different answers, the proposed system will choose the one having the best performance at the offline training stage.Fig. 41 shows the flow chart of the proposed framework while Fig. 1 shows the final results.

IX. RESULTS AND LESSONS LEARNED
In Track 3 Smartphone 11 teams were registered and submitted their final results.9 out of 11 teams scored under 7 m, with the best one at 1 m.Despite the challenges imposed by the evaluation scenario, with some open areas and a few unmapped locations, 4 teams achieved a positioning error lower than 2 m.Competitors had a few months to process the pre-collected data.The key to success is the sensor fusion approach, which was adopted by all systems which obtained a score below 3 m.Kalman Filter (KF) and all its variants are very popular for this task.
In Track 4 Foot-mounted IMU 5 teams were registered, of which 4 accepted to publish their results.There were two final scores under 7 m.What was really amazing is the score reached by the winner: 0.5 m on such a scenario is impressive.The key of success is an excellent implementation of ZUPT.Techniques seem to be really up to scratch now, at least on scenarios with a constant walking pattern.For IPIN 2021, Track 4 plans to add a running pattern for a more challenging competition.
In Track 5 xDR in manufacturing 4 teams registered and 2 submitted their final results.Final scores (CE75) of the Track 5's winners is higher than in other Tracks.Considering the fact that the data are measured in actual industrial situation, the achieved results in PDR-subtrack can to be regarded as positive.
The results in the VDR sub-track are worse than expected, exceeding 7 m.One possible reason is the lack of BLE beacons in the area of the VDR sub-track.Another one may be the lack of awareness surrounding VDR methods: maybe educational campaigns are needed to spread knowledge about VDR methods among researchers and practitioners.
In Track 6 On-vehicle smartphone 3 teams were registered and submitted their final results.Two final scores were under 30 m, with the best one at 7 m.The key of success is the perfect use of vehicle motion constraint information and magnetometer observations including ZUPT, ZIHR, Non-Holonomic Constraints (NHC) and magnetic heading.Considering the long interruptions of GNSS signal in the test data, more to the point is to maintain the vehicle heading accurate.
In Track 7 Channel impulse response only one team participated in the competition, reaching an evaluation score of 1.4 m.We expect future editions to see more widespread participation in this Track which is focused on a still littleknown, leading-edge method.

A. Lessons Learned
Maybe the most important results of the IPIN competition are comments made by competitors and observations made by Track chairs about the competing systems performance.Here we summarise the most important ones.

1) Competitor Observations:
• Some competitors reported that the heading estimation is a critical step for systems based on sensor fusion.It seems that it only might work well when the smartphone is held in front of the body.This was mostly the case in the evaluation trajectory, except for one relatively short time interval.To improve accuracy, it is necessary to use a PDR algorithm that can better handle realistic movement and different walking modes.Thus, more phone carrying modes are required both at calibration and evaluation phases to handle realistic scenarios.• Floor plans are essential for map-based localisation approaches.It may be beneficial for such competitors to have access to additional information about the building, i.e., pictures, videos, etc.The better understanding of the building with its specifics leads to more informed decisions regarding the system components and parameter configurations.
• Some competitors used visual inspection to choose the final submitted trajectories.This approach is not straightforward when comparing two trajectory candidates without knowing the ground truth locations of the evaluation points.Although it was quite feasible to identify entry and exit positions for the floor transition in the estimated trajectories, the path accuracy especially in larger open areas was difficult to rate.After the competition, knowing the ground truth location of the evaluation points, competitors realised that more rigorous methods for trials comparison are needed instead of relying in simple visual consideration.how to analyse the foot-mounted IMU's signal characteristics in two unique scenes (escalator or lift) as well as the pedestrian positioning algorithms in these two unique indoor environments.Because the IMU noise is different between dynamic and static conditions, it was possible to fine-tune the sensor parameters based on the Allan variance.The adjustment method is to meet the optimal result of zero-speed correction under long-term static data.Moreover, some re-visits of the trajectory were found and made use of such valuable opportunities to correct the drift of foot-INS through close loop adjustment (smoothing like Simultaneous Localization and Map (SLAM)).How to improve the stability and reliability of Foot-INS is also an important issue.
2) Track Chairs Observations: • Regarding Track 3 Smartphone, this is the fifth year in a row using the same data collection strategy and format to store the data.Despite that, it does not lose interest from the research community and has achieved gradual improvements in results year after year, showing that Track 3 is very competitive, and research teams are still interested in participating.Some teams have reported that this Track, with the collected data sets, have allowed them to improve their systems year by year.• In Track 4 Foot-mounted IMU, this year we witnessed a wide range of resulting performance, with the winner doing much better than the other competitors.That means that in the future Track 4 competition will have to be more competitive and at the same time will have to pay attention to let the doors open to new competitors.One possible solution would be to add complex pattern like running or jumping; organisers also envisage to use a novel sensor delivering barometer data in addition to GNSS, IMU and magnetometer signals.
• For Track 5 xDR in manufacturing, we have kept challenging new trails in evaluation.Due to the difficulty in the realistic scenarios and the novelty of the competitions, some teams gave up before submitting the results; as a result, the number of the participants was less than expected.In order to attract more competitors, it is better to provide chances for using the evaluation framework and evaluation indicators.Sharing the evaluation scripts on GitHub will help promoting the evaluation framework.Moreover, development of the VDR method should be encouraged for boosting competitions of VDR.• This was the debut year for Track 6 On-vehicle smartphone.Performance of competing system was widely varied, with the top two teams at about 10 n, which is within the expected range.In IPIN 2021 an odometer sensor could be added to make it closer to the vehicle scene.At the same time, considering that more and more mobile phones can support differential positioning, differential positioning results will be provided to improve positioning accuracy.In addition, changing the posture of the mobile phone during the test will be considered, as this is a typical case in the real scene.• Track 7 Channel impulse response is based on CIR, which is a novel topic and as such has not attracted many competitors.Since, in terms of RF signalling, most of the indoor positioning community is focusing on RSS-or range-based methods, a more detailed description of CIRand other channel-based methods, including an extensive reference to recently published related approaches could have made the implied positioning task clearer.Furthermore, an example processing pipeline could have provided more guidance in handling the data.The amount of training data could have been reduced to allow for less computationally demanding computations.In the end, we believe that interest in this are is bound to grow with time, given its great promises.
X. CONCLUSION The IPIN Competition has been highly relevant for the indoor positioning community since the first edition held in Busan (Korea) in 2014.Based on the EvAAL framework, the purpose of IPIN competitions is to evaluate positioning solutions from academy and industry in challenging environments, using realistic procedures on a level field.
For the first time, the 2020 edition did not host on-site Tracks (Tracks 1 and 2), because of worldwide travel restrictions.However, the number of off-site Tracks was a record-high.The evaluation areas included a library building, a shopping mall, an indoor-outdoor road Track and industrial-oriented environments.
Of the 21 teams competing online in 2020 in Tracks 3-7, 20 accepted to contribute to this paper and concisely described their algorithm workflow.This collection is arguably the best description we can get today of state-of-the-art in personal indoor localisation systems at the algorithmic level.
21 competing teams and 95 attendants to the final online event witness a vibrant activity in the personal positioning field.This activity is focused on creating an environment where indoor localisation systems can provide general and cheap ways to position, track and navigate people indoors as easily as GNSS does outdoors.
Most competitors of smartphone-based systems have employed as many sources of information as possible for positioning, making it clear that not only good sensing capabilities are relevant for positioning, but also rich context information (maps, images, videos) plays a key role in enhancing positioning accuracy.
In the previous section we have presented a short overview of results.We find them impressive, especially with respect to what was available just few years ago.
Yet, these observed results highlight a significant gap between the accuracy reported in the literature and the results obtained in the competition.It is far too easy to find accuracies reported in the literature which are unrealistically good with respect to what we observe in on-site Tracks.What is more uncomfortable is finding the same even with respect to off-site Tracks, which generally provide far better results.This is mostly due to insufficient test and evaluation procedures, as the vast majority of papers in the literature present results obtained by simulation or trial in a small lab.Some papers present results obtained in larger areas (usually one floor of an office or university building) with an actor walking at a natural pace.Still, very few papers that we know of consider testing in unfamiliar areas, thus minimising the effect of building a system tuned to the laboratory environment.
Indoor localisation and seamless location-based services are enablers for an enormous market that will develop in the near future.The IPIN competitions have played an essential role in the academic and industrial research in this field; as far as we can tell they are going to play it for the foreseeable future as well.

CONTRIBUTIONS
The IPIN 2020 Competition was organized by ISTI in Pisa, an institute of the CNR (National Research Council) of Italy.Francesco Potortì chaired the competition and was responsible for the final overall review of the paper.
The team managing Track 4 (foot-mounted IMU-based positioning) was composed by Miguel Ortiz, Ni Zhu, and Valerie Renaudin.
The team managing Track 6 (smartphone-based vehicle positioning without additional equipment) was composed by Dongyan Wei, Xinchun Ji, and Wenchao Zhang.
The team managing Track 7 (channel impulse responses) was composed by Sebastian Kram, Maximilian Stahlke, and Christopher Mutschler.
The team managing the online event competition was composed by Antonino Crivello, Paolo Barsocchi, Michele Girolami, and Filippo Palumbo.

Fig. 3 .
Fig. 3. Tasks during the training stage of WHU-FIVE team system.
a) Training stage: In the training stage, four tasks are performed that can be seen in Fig. 3.The first one is the feature map construction.In this task, the Wi-Fi fingerprint and geo-magnetic fingerprints map is built, and extract the book-shelf layout from the book-shelf training data set.The second task is the training of the step length model.The third one is the extraction of the stair-steps' number of each two layers from the Floor-Transition data set.The last task is training the motion pattern model to recognize some motion types, such as up/down stairs, in/out doors, in-situ steering motion and phone-holding posture.

Fig. 8 .
Fig. 8. Magnetic field based localization from the NLE team system.Magnetic field data captured by the mobile phone are transformed in time series.Encoded into 2D images, they form an input for Convolutional neural network (CNN) and Recurrent Neural Network (RNN) trained to predict the user's position.

Fig. 11 .
Fig. 11.The membership functions used by the YAI team system for the received RSS values.

Fig. 12 .
Fig. 12.The defuzzification process used by the YAI team system.

Fig. 13 .
Fig.13.The resulting CDF of the positioning errors with respect to the testing data set with the YAI's system.

Fig. 22 .
Fig. 22.The scheme of foot-mounted PDR system based on multi-constraint algorithms proposed by AIR team.

Fig. 23 .
Fig. 23.Revise the current step's inertial recursive position with the position calculated from the stride heading in the AIR team system.

Fig. 24 .
Fig. 24.System architecture of proposed pedestrian inertial navigation based on motion mode recognition proposed by Free-Walking team.

Fig. 25 .
Fig. 25.Pedestrian walking mode recognition based on the stacked denoising autoencoder and temporal convolutional network with attention in the Free-Walking team system.

Fig. 31 .
Fig. 31.The scheme of three steps indoor localization system of KawaguchiLab team system.

Fig. 32 .
Fig. 32.The test route and GNSS condition of Track 6.

Fig. 34 .
Fig. 34.Flowchart of the vehicle integrated positioning algorithm based on smartphone built-in sensors of the WHU&Autonavi Team system.

Fig. 35 .
Fig. 35.The RNN framework of the proposed method by ZU-Mellivora Capensis team.

Fig. 36 .
Fig. 36.Flow chart for the proposed tracking framework of YAI team system.

Fig. 37 .
Fig. 37.The estimated driving direction during the testing period, where the horizontal axis is time with the unit second and the vertical axis is the estimated angle.

Fig. 38 .
Fig. 38.Measurement setup.The industrial environment consists of metal shelves and industrial vehicles.The Receiver/Anchor tags are highlighted with red boxes, the transmitter/mobile node is highlighted with a green circle.

Fig. 39 .
Fig. 39.Data distribution of the recorded data set.The objects in the environment are indicated in green: Metal shelves filled with goods (1) and (3); industrial vehicles (2) and (4) and a large metal box(5).The anchor node positions are depicted as red squares.

Fig. 40 .
Fig. 40.The network architecture of the used DNN for CIR-based positioning of YAI team proposed system.

Fig. 41 .
Fig. 41.Flow chart of the system proposed by YAI team for CIR-based positioning.

TABLE II RESULTS
FOR ALL TRACKS.THE FIRST COLUMN IS THE COMPETITION SCORE (EQUATION 2), WHILE THE REMAINING COLUMNS SHOW OTHER COMPLEMENTARY RELEVANT METRICS (MEAN, RMSE.MEDIAN, 95 th PERCENTILE AND FLOOR HIT RATE (IF AVAILABLE).WE ALSO INCLUDE A REFERENCE TO THE SECTION WHERE THE SYSTEM IS DESCRIBED Fig. 1.Cumulative distributions of point errors (Equation 1).

TABLE V STEPS
DESCRIPTION COMPOSING Data Set Num 2

•
Deep learning requires large scale annotation to train accurate models.As the challenge data is sparsely annotated, one can proceed by pseudo-labelling non-annotated sensor data.This weak annotation works well in narrow corridors where the user's position can be accurately approximated.However, in open spaces, like in Track 3 Smartphone, weak annotations are harder to get straight.Approximation of user's position is less accurate and this added noise hurts the performance of deep learning models.This raises the issue of finding alternate ways to densely annotate sensor data.•In Track 4 Foot-mounted IMU, there are some special scenes such as carousels, frequent stair, elevator switching as well as frequent pedestrian walking modes switching.Therefore, designing an accurate real-time algorithm to recognise localisation environment and walking mode is essential for foot-mounted pedestrian positioning.