Translating Agent-Environment Interactions from Humans to Robots

Humans are remarkably adept at imitating other people performing tasks, afforded by their ability to abstract away irrelevant details and focus on the task strategy of the demonstrator. In this paper, we take steps towards enabling robots with this ability, and present a framework, TransAct to do so. TransAct first builds on prior skill learning work to learn temporally abstract representations of common agent-environment interactions in manipulation tasks, e.g., a robot pouring from a cup. Given a human demonstration of an unseen unknown task, TransAct then translates the underlying sequence of interactions (i.e., the human task strategy) to a robot learner. Through experiments on real-world human and robot datasets, we demonstrate TransAct’s ability to accurately represent diverse agent-environment interactions. Moreover, TransAct empowers robots to consume human task demonstrations and compose corresponding interactions with similar environmental effects to perform the tasks themselves in a zero shot manner, without access to paired demonstrations or dense annotations. We present visualizations of our results at https://sites.google.com/view/interaction-abstractions.