Nowadays rapid and accurate speech retrieval techniques based on semantics are desired for the overwhelming amounts of speech data. In this paper we mainly study a converted lattice-based approach for Chinese spontaneous speech retrieval. A new confidence measure method is proposed based on context mutual information. In our knowledge, it is firstly used in a lattice construction for speech indexing. The method can take full advantage of the mutual information between words in order to describe the language model more exactly. Our experiment results show that the proposed method in this paper outperforms both posterior probability based method and N-best based method. And our best system achieves a FOM of 81.2% on a task of spontaneous Chinese speech retrieval.