Recently, remote user authentication schemes are implemented on elliptic curve crypto-System (ECC) to reduce the computation loads for mobile devices. However, most remote user authentication schemes on ECC are based on public-key cryptosystem, in which the Public key in the system requires the associated certificate to prove its validity. Thus, the user needs to perform additional computations to verify the certificate in these schemes. In addition, we find these schemes do not provide mutual authentication or a session key agreement between the user and the remote server. Therefore, we propose an ID-based remote mutual authentication with key agreement scheme on ECC in this paper. Moreover, we adopt multi-servers to realize the scheme's scalability. Compared with the related works, the proposed scheme is more efficient and practical for distributed or portable devices.