Imitating human joystick control ability using style and content disentanglement

Ability to extract user’s joystick input style from a few demonstrations and applying it to various navigation tasks can be useful for automated testing of shared control algorithms in simulation. This can reduce the need for user studies, leading to time and cost benefits, especially when human subjects suffer from disabilities. State-of-the-art imitation learning methods require demonstration for every task making them unsuitable for this use case. Methods like adversarial motion prior (AMP) and learning from play (Play-LMP) provide an alternative. However, data used to train AMP’s discriminator is generated by task-specific policies which can limit its ability to apply a style to other tasks. Also, it tries to optimize both style and task reward simultaneously which can lead to style not being imitated in favor of task completion. Instead of learning style through a discriminator, Play-LMP tries to extract style by learning latent representations. However, this is done in an unsupervised manner making it hard to enforce the correlation between the learned latent representation and the user’s input style. In this work, we investigate how the supervised latent optimization based disentanglement approach (used to disentangle images to its style and content latent), can be used to extract style from a few human demonstrations and applied to different tasks with simulator in the loop. Our initial results on synthetic data show that this can be a promising approach.