A General Approach for the Automation of Hydraulic Excavator Arms Using Reinforcement Learning

This article presents a general approach to derive an end effector trajectory tracking controller for highly nonlinear hydraulic excavator arms. Rather than requiring an analytical model of the system, we use a neural network model that is trained based on measurements collected during operation of the machine. The data-driven model effectively represents the actuator dynamics including the cylinder-to-joint-space conversion. Requiring only the distances between the individual joints, a simulation is set up to train a control policy using reinforcement learning (RL). The policy outputs pilot stage control commands that can be directly applied to the machine without further fine-tuning. The proposed approach is implemented on a Menzi Muck M545, a 12 $\mathrm{t}$ hydraulic excavator, and tested in different task space trajectory tracking scenarios, with and without soil interaction. Compared to a commercial grading controller, which requires laborious hand-tuning by expert engineers, the learned controller shows higher tracking accuracy, indicating that the achieved performance is sufficient for the practical application on construction sites and that the proposed approach opens a new avenue for future machine automation.