Learning to Adjust and Refine Gait Patterns for a Biped Robot

In this paper, a reinforced learning method for biped walking is proposed, where the robot learns to appropriately modulate an observed walking pattern. The biped robot was equipped with two Q -learning mechanisms. First, the robot learns a policy to adjust a defective walking pattern, gait-by-gait, into a more stable one. To avoid the complexity of adjusting too many joints of a humanoid robot and to speed up the learning process, the dimensionality of the action space was reduced. In turn, the other learning mechanism trained the robot to walk in a refined pattern, allowing it to walk faster without the loss of other required criteria, such as walking straight. This approach was implemented with both a simulated robot model and an actual biped robot. The results from the simulations and experiments show that successful walking policies were obtained. The learning system works quickly enough so that the robot was able to continually adapt to the terrain as it walked.

I. Introduction

Recently, due to the fast development of innovative techniques in computers, sensors, and motors, biped locomotion has attracted a lot of research interest. There are different methodologies for tackling the problem of gait pattern generation and walking balance for biped robots. In static walking approaches [1], the control architecture has to ensure that the projection of the center of gravity (CoG) on the ground is always inside the support polygon (SP), which is formed by the floor support points. Static walking assumes that a biped robot is statically stable at any time and stays indefinitely in a stable position if all motions are stopped. Therefore, the projection of the robot’s CoG on the ground must be contained within the SP. This approach was abandoned because it could only achieve slow walking speeds on flat surfaces. Dynamic stable walking approaches [2]–[4] allow the projection of the CoG to be outside the SP, but the zero moment point (ZMP), where the robot’s total moment in the horizontal direction at the ground is zero, cannot move out of the SP. A dynamically walking biped only allows the projection of the CoG outside the SP for limited amounts of time, as long as the robot can always keep at least one foot flat on the ground. Meanwhile, dynamic walking requires that the robot always rotates around a point in the SP. The robot rotates around a point outside the SP only when the supporting foot is going to leave the ground or is about to be pressed against the ground. If all motions are stopped, the robot tends to rotate around the ZMP. The position of the ZMP, which can be used as a stability criterion, is computed by finding the point on the foot where the total torque is zero.