Multi-Modal Generative Models for Learning Epistemic Active Sensing

We present a novel approach of multi-modal deep generative models and apply this to coordinated heterogeneous multi-agent active sensing. A major approach to achieve this objective is to train a multi-modal variational Auto Encoder (M
²
VAE) that integrates the information of different sensor modalities into a joint latent representation. Furthermore, we derive an objective from the M
²
VAE that enables the maximization of the evidence lower bound via selection of sensor modalities. Using this approach as a direct reward signal to a multi-modal and multi-agent deep reinforcement learning setup leads intuitively to an epistemic active sensing behavior that coordinately resolves the ambiguity of observations.