In this paper, we propose a general two-dimensional hidden Markov model (2D-HMM), where dependency of the state transition probability on any state is allowed as long as causality is preserved. The proposed 2D-HMM model can capture, for example, dependency among diagonal states, which can be critical in many image processing applications. A new expectation-maximization (EM) algorithm suitable for estimation of the new model is derived, where a novel general forward-backward (GFB) algorithm is proposed for recursive estimation of the model parameters. A new conditional-independent subset-state sequence structure decomposition is proposed for the 2D Viterbi algorithm. The new model can be applied to many areas such as trajectory classification and image segmentation. Application to aerial image segmentation shows the superiority of our model compared to the existing 2D-HMM model.