Vision-Based Excavator Control for Pick-and-Place Operation*

Automating an excavator in construction tasks proves to be challenging. While conventional kinematics control can be employed to produce motion, the challenge lies in the perception task: how to distinguish and detect various objects in construction sites. These include construction objects (rocks, sands, etc) as well as vehicles for object storage. This paper proposed vision-based excavator control combining deep learning and inverse kinematics to perform pick-and-place operation. The perception side is handled by a convolution neural network to detect target object and vehicle for object storage. Combined with depth information, the location of the objects are determined. This information will be sent to the control side to calculate the target joints of the excavator using inverse kinematics. Finally, joint velocity control is employed to ensure the excavator's joints reach the target. The results confirm that the proposed control successfully performs autonomous pick and place in simulated environment.