In visual servo control of a robot, we often encounter the structure-from-motion problem. To study the structure-from-motion problem, we are led to finding a minimum of a real valued function defined on a product Riemannian manifold, e.g., special orthogonal groups and unit sphere. To take advantage of its Riemannian structure, we consider the Newton algorithm on this manifold. In particular, we focus on improving the algorithm to be more robust and faster than the existing Newton algorithm on Riemannian manifolds. For this, we exploit the sparseness of the Hessian matrix and suggest how to choose the step size during the optimisation procedure, which can be considered as extensions of those for vector space optimisation algorithms.