Temporal Logic Guided Affordance Learning for Generalizable Dexterous Manipulation

Reinforcement learning has shown great potential to address long-horizon tasks. However, most existing work lacks the ability to reason about functional parts of objects and extract the task semantics to facilitate robotic learning, making it difficult to achieve category-level generalization even on seen objects. To address these issues, we develop a temporal logic guided affordance learning framework for generalizable dexterous manipulations (TALD). In particular, TALD is equipped with affordance learning for predicting actionable information and LTL representation for understanding the task semantics and improving learning efficiency. We design a contact predictor from the affordance learning perspective to improve the generalization performance, which predicts the max-affordance point observation based on the 3D point clouds to guide the agent to manipulate the functional part of objects. And we exploit the LTL progression to construct a task-driven labeled POMDP to address the challenge of a non-Markovian reward function, and design a task module to extract the LTL representation by Transformer Encoder to improve the sampling efficiency and facilitate the robotic learning. We validate the proposed method in four articulated manipulation scenarios. The generalization performance corresponding to success rates and visualization effects show the effectiveness of TALD. Our project is available at: https://sites.google.com/view/tald-0257/.