In this paper, we document the progress in the design of a motion segmentation and control strategy for a smart assistive robot arm that can provide assistance during activities of daily living to the elderly and/or users with disabilities. Interaction with the environment is made challenging by the kinematic uncertainty in the robot, imperfect sensor calibration as well as the fact that most activities of daily living are generally required to be performed in unstructured environments. The motion control strategy exploits visual and force feedback from sensors in the robot’s hand to provide the basis for efficient interaction with the unstructured environment. Through experimental studies with a variety of objects of daily life in natural environments, an anthropomorphic-like approach was found to be the most suitable for reliable and speedy object retrieval. Specifically, gross reaching/docking motions of the robot arm using proprioception are followed by fine alignment of the hand through visual feedback and eventually grasping based on haptic feedback. Experimental results using a wheelchair mounted robotic arm are presented to demonstrate the efficacy of the proposed algorithms.