Autonomous Elevator Button Recognition and Operation Framework for Multi-Floor Mobile Manipulator Navigation

A wheeled mobile manipulator to navigate between floors needs to recognize the information of the elevator’s buttons and operate them. This paper proposes an autonomous elevator button recognition and operation framework for multi-floor mobile manipulator navigation. The proposed framework consists of three sequential phases, the detection of the ROI, the recognition of numeral texts and the operation of the elevator’s buttons. In Phase I, a connective-region-based method is proposed to locate the buttons and to detect the numeral texts. In Phase II, a small-scale convolutional neural network model is proposed to recognize the numeral texts. In Phase III, an error correction mechanism based on the spatial arrangement of the buttons and the continuity of numbers is designed to eliminate the false positive detection and recognition results. The proposed framework is evaluated by dataset consisting of images of various elevators’ button panels. The results show that the proposed framework is better than the state-of-the-art method in terms of the accuracy and the robustness when faced with different elevators. The proposed framework is also evaluated in real-world experiments on a mobile manipulator.