Combining Model Predictive Path Integral with Kalman Variational Auto-encoder for Robot Control from Raw Images

In this paper, we propose a framework for model learning and control of complex non-linear systems from such high-dimensional and redundant sensor observations as raw pixel images. Our framework combines Kalman variational auto-encoder (KVAE) and model predictive path integral (MPPI) to exploit both advantages of the efficient learning and inference algorithms thanks for the simple structure of latent dynamics in KVAE and sampling-based derivative-free computation of control in MPPI. We applied our framework for an air hockey task in simulations, and simulation results verified its effectiveness for model predictive control from the raw images capturing the task environment as observations without directly observing the states in the controller.