Although considerable effort has been devoted to both fundamental frequency (F0) and spectral envelope estimation in the field of speech processing, the problem of determining F0 and spectral envelopes has largely been tackled independently. If F0 were known in advance, then the spectral envelope could be estimated very reliably. On the other hand, if the spectral envelope were known in advance, then we could obtain a reliable F0 estimate. F0 and the spectral envelope, each of which is a prerequisite of the other, should thus be estimated jointly rather than independently in succession. On this basis, we develop a parametric speech spectrum model that allows us to estimate the F0 and spectral envelope simultaneously. We confirmed experimentally the significant advantage of this joint estimation approach for both F0 estimation and spectral envelope estimation.