Morphological building index (MBI) is a recently developed approach for automatic indication of buildings in high-resolution imagery. However, MBI is subject to commission errors due to the similar characteristics between buildings, bare soil and roads. Furthermore, omission errors occur in dark and heterogeneous roofs. In this study, a systematic framework for building extraction from high-resolution imagery is proposed, aiming to alleviate both commission and omission errors for the original MBI algorithm. The improvements include three aspects: 1) a morphological shadow index (MSI) is proposed to detect shadows that are used as a spatial constraint of buildings; 2) a dual-threshold filtering is proposed to integrate the information of MBI and MSI; 3) the proposed framework is implemented in an object-based environment, where a geometrical index and a vegetation index are then used to remove noise from narrow roads and bright vegetation. The proposed framework was validated on an Ikonos image of Washington DC Mall with 1-m resolution and an 8-channel WorldView-2 image of Hangzhou, east of China, with 2-m resolution. By comparison with the ground truth references, it was shown that our method achieved over 90% overall accuracy for discrimination between buildings and backgrounds for both datasets. In the comparative study, it was revealed that the proposed method improved the original MBI significantly. Furthermore, the proposed method was more accurate than the support vector machine interpretation with the differential morphological profiles (DMP) and multiscale urban complexity index (MUCI).