Reinforcement learning for quadruped robot locomotion control

Abstract

Quadruped robots are receiving more attention from academics and industry due to their excellent dynamic locomotion capabilities. Many complex tasks are being explored to be completed by quadruped robots, which also brings challenges with increasing performance requirements. Traditional model predictive control has been proven to have significant effects on quadruped robots. Reinforcement learning has also shown strong potential in controlling flexibility and robustness in quadruped robots. Therefore, this paper mainly discusses the control of quadruped robots in complex terrain using reinforcement learning methods based on model predictive control. The key point of the quadruped robot locomotion control based on model predictive control is to transform the prediction of the future multiple steps into an optimization problem. The author analyzes its force condition through a single rigid body model and lists its state equation and prediction equation. The control problem is transformed into an optimization problem using the prediction equation to calculate the plantar reaction force. Finally, the control command is solved by combining whole-body control. This solution is verified by building a physical quadruped robot. Compared with model predictive control methods, reinforcement learning methods learn the optimal policy by simulating the interaction between the robot and the environment in simulation and maximizing the cumulative rewards. This thesis uses the Isaac Gym platform which can simulate thousands of robots at the same time by the parallel computing characteristics of GPU. The proximal policy optimization algorithm is used to train the policy. In combination with the robot model, the author adjusts the reinforcement learning hyperparameters and observations and sets new tasks and corresponding reward functions to obtain policies suitable for complex and challenging terrains. Due to the error between the physical robot and the simulation model, an unsupervised learning method, multi-layer perceptron, was used to train the actuator network, which rebuilds the motor model, to solve the sim to real problem. Through the actuator net, the author further verifies the robustness and flexibility of the controller based on reinforcement learning on the physical robots. This was verified in complex and challenging indoor and outdoor terrains and international competition.