Multi-Hand Direct Manipulation of Complex Constrained Virtual Objects

We present a method to manipulate virtual objects which are constrained complexly and reconfigurable as if they would exist in a real world by using multiple hands simultaneously. A complexly constrained, and reconfigurable object, such as a Rubik's cube, is hard to describe its physical motion constraints, mainly because they are determined by the grasping situation and dynamically changeable. Rather than describing the physical motion constraints in a general form, we more focus on the multiple hand interaction of a complex object. A complex object is divided into multiple subparts which are grasped by each hand, and the constraints between the subparts are optimized for inducing natural and continuous movement. For this, we propose a dynamically adjustable data structure for representing object parts grasped by multiple hands, and an optimization-based pose estimation of the constrained subparts along with their grasped hands. The experiments show that human subjects can manipulate a complexly constrained object such as a Rubik's cube without any difficulty as if it exists in the real-world.