Learning to Play Table Tennis From Scratch Using Muscular Robots

Dynamic tasks such as table tennis are relatively easy to learn for humans, but pose significant challenges to robots. Such tasks require accurate control of fast movements and precise timing in the presence of imprecise state estimation of the flying ball and the robot. Reinforcement learning (RL) has shown promise in learning complex control tasks from data. However, applying step-based RL to dynamic tasks on real systems is safety-critical as RL requires exploring and failing safely for millions of time steps in high-speed and high-acceleration regimes. This article demonstrates that using robot arms driven by pneumatic artificial muscles (PAMs) enables safe end-to-end learning of table tennis using model-free RL. In particular, we learn
from scratch
for thousands of trials while a stochastic policy acts on the low-level controls of the real system. The robot returns and
smashes
real balls with
$5 $
ms
⁻¹
and
$12 $
ms
⁻¹
on average, respectively, to a desired landing point. Additionally, we present hybrid sim and real training (HYSR), a practical procedure that avoids training with real balls by virtually replaying recorded ball trajectories and applying actions to the real robot. To the best of authors’ knowledge, this work pioneers (i) failsafe learning of a safety-critical dynamic task using anthropomorphic robot arms, (ii) learning a precision-demanding problem with a PAM-driven system that is inherently hard to control as well as (iii) train a robot to play table tennis without real balls.