An adaptive neural network controller is proposed to deal with the task-space tracking problem of manipulators with kinematic and dynamic uncertainties. The orientation of manipulator is represented by the unit quaternion, which avoids singularities associated with three-parameter representation. By employing the adaptive Jacobian scheme, neural networks, and backstepping technique, the torque controller is obtained which is demonstrated to be stable by the Lyapunov approach. The adaptive updating laws for controller parameters are derived by the projection method, and the tracking error can be reduced as small as desired. The favorable features of the proposed controller lie in that: (1) the uncertainty in manipulator kinematics is taken into account; (2) the unit quaternion is used to represent the end-effector orientation; (3) the "linearity-in-parameters" assumption for the uncertain terms in dynamics of manipulators is no longer necessary; (4) effects of external disturbances are also considered in the controller design. Finally, the satisfactory performance of the proposed approach is illustrated by simulation results on a PUMA 560 robot.