This paper presents a new approach that uses the maximum model distance (MMD) method for the adaptation of hidden Markov models (HMMs). This method has the same framework as it is used for constructing speech recognizers with abundant data, and work effectively with any amount of adaptation data. All parameters of the HMMs with or without the adaptation data could be adapted. If the adaptation data is sufficient, then the adapted models will gradually become a speaker-dependent one. Both the dialect and the speaker adaptation experiments were conducted to investigate the effectiveness of the proposed algorithm. In the speaker adaptation experiments, up to 65.55% phoneme error reduction was achieved, and the MMD could reduce the phoneme error by 16.91% even only one adaptation utterance is available.