Semantic Map-Based Visual Localization With Consistency Guarantee

In this article, a semantic map-based consistent extended Kalman filter (EKF) for drift-free visual localization is presented. Thanks to the advanced deep learning techniques, object-level or semantic-level information in the scene can be considered in the localization task. However, direct fusion of object pose observations using EKF suffers from inconsistency problems, leading to false information along with unobservable directions. Considering the natural symmetric group structure of the localization system, we properly define invariant errors with intrinsic consistency properties. This is accompanied by strictly linear error propagation, which avoids the introduction of linearization errors. In the observability analysis, it is proven that the proposed algorithm can automatically maintain the correct unobservable subspace. Moreover, detailed algorithmic descriptions of state propagation, measurement update, and state augmentation are also presented. The performance and consistency of the proposed method are evaluated and compared via extensive Monte-Carlo simulations and real-world experiments.