SSFold: Learning to Fold Arbitrary Crumpled Cloth Using Graph Dynamics From Human Demonstration

Robotic cloth manipulation poses significant challenges due to the fabric’s complex dynamics and the high dimensionality of configuration spaces. Previous approaches have focused on isolated smoothing or folding tasks and relied heavily on simulations, often struggling to bridge the sim-to-real gap. This gap arises as simulated cloth dynamics fail to capture real-world properties such as elasticity, friction, and occlusions, causing accuracy loss and limited generalization. To tackle these challenges, we propose a two-stream architecture with sequential and spatial pathways, unifying smoothing and folding tasks into a single adaptable policy model. The sequential stream determines pick-and-place positions, while the spatial stream, using a connectivity dynamics model, constructs a visibility graph from partial point cloud data, enabling the model to infer the cloth’s full configuration despite occlusions. To address the sim-to-real gap, we integrate real-world human demonstration data via a hand-tracking detection algorithm, enhancing real-world performance across diverse cloth configurations. Our method, validated on a UR5 robot across six distinct cloth folding tasks, consistently achieves desired folded states from arbitrary crumpled initial configurations, with success rates of 100.0%, 100.0%, 83.3%, 66.7%, 83.3%, and 66.7%. It outperforms state-of-the-art cloth manipulation techniques and generalizes to unseen fabrics with diverse colors, shapes, and stiffness. Project page: https://zcswdt.github.io/SSFold/ Note to Practitioners—In this paper, we introduce SSFold, a novel framework for robotic cloth manipulation that integrates human demonstrations with advanced learning techniques, providing a practical solution for real-world applications. Practitioners in industries such as textile manufacturing, automated laundry services, and even medical fabric handling can leverage this method to improve operational efficiency and reduce reliance on manual labor significantly. By using a two-stream architecture to handle complex cloth dynamics and self-occlusions, SSFold unifies smoothing and folding tasks into a single policy model, which adapts effectively to diverse fabric types and conditions. A key advantage of this method lies in its ability to utilize low-cost, easy-to-set-up hand tracking systems for human demonstration data collection, reducing the need for expensive, complex setups. The framework has been successfully validated in real-world scenarios with a UR5 robot, achieving high success rates across a variety of folding tasks. Furthermore, the method demonstrates strong generalization capabilities, making it suitable for a wide range of applications beyond the tasks it was initially trained on. This scalability and flexibility offer a highly practical and cost-effective solution for integrating robotic cloth manipulation into various industries.

I. Introduction

Cloth manipulation has a wide range of applications in both domestic and industrial settings, such as laundry unfolding [1] and folding [2], surgery [3], and manufacturing [4]. These applications enhance the quality of life by reducing human labor. However, it has posed a challenge for robotic manipulation: compared to rigid objects, cloth has infinite degrees of freedom, can be only partially observable due to self-occlusions in crumpled configurations, and does not transform rigidly when manipulated. The dynamics of cloth are also complex [5], and slightly different interactions may lead to significantly different cloth behaviors. Early approaches for cloth manipulation efforts relied heavily on scripted actions, which were generally slow and lacked the flexibility to adapt to varying cloth configurations.