Structure-Aware Policy to Improve Generalization among Various Robots and Environments

Recently, Deep Reinforcement Learning (DRL) has been used to solve complex robot control tasks with outstanding success. However, previous DRL methods still exist some shortcomings, such as poor generalization performance, which makes policy performance quite sensitive to small vari-ations of the task settings. Besides, it is quite time-consuming and computationally expensive to retrain a new policy from scratch for new tasks, hence restricts the applications of DRL-based methods in the real world. In this work, we propose a novel DRL generalization method called GNN-embedding, which incorporates the robot hardware and the environment simultaneously with GNN-based policy network and learnable embedding vectors of tasks. Thus, it can learn a unified policy for different robots under different environment conditions, which improves the generalization performance of existing DRL robot policies. Multiple experiments on the Hopper-v2 robot are conducted. The experimental results demonstrate the effectiveness and efficiency of GNN-embedding on generalization, including multi-task learning and transfer learning problems.

I. Introduction

With the growing interest and development in deep rein-forcement learning (DRL), we have seen remarkable progresses and achievements of DRL in various application scenarios, including video games [1], autonomous vehicles [2], and robotics control [3]–[5]. Despite the huge success, existing DRL-based methods are still quite limited on generalization [6], which restricts real applications of DRL methods, especially in robot control tasks. The policy highly depends on the parameter settings of the task, thus it can only learns how to control a single robot in a single environment at one time. For example, given robot control tasks with different robot hardware implementations (link length, etc.) and environment features (friction coefficient, etc.), we are required to train multiple policies, despite the similarity among these tasks, which is quite computationally expensive and time-consuming. Therefore, it is essential to design a unified DRL method, which can be utilized across various robots and environments simultaneously. Fig. 1.

The visualization of the generalization task and GNN -embedding composition. The task pool k contains robots with different hardware parameters and environments with various characteristics. Each task is a combination of a robot and an environment. Gnn -embedding is a unified policy to control multiple tasks, which is able to learn both the task-invariant knowledge and the task-specific knowledge.