Since remote sensing provides more and more sensors and techniques to accumulate data on urban regions, three-dimensional representations of these complex environments gained much interest for various applications. In order to obtain three-dimensional representations, one of the most practical ways is to generate Digital Surface Models (DSMs) using very high resolution remotely sensed images from two or more viewing directions, or by using LIDAR sensors. Due to occlusions, matching errors and interpolation techniques these DSMs do not exhibit completely steep walls, and in order to obtain real three-dimensional urban models including objects like buildings from these DSMs, advanced methods are needed. A novel approach based on building shape detection, height estimation, and rooftop reconstruction is proposed to achieve realistic three-dimensional building representations. Our automatic approach consists of three main modules as; detection of complex building shapes, understanding rooftop type, and three-dimensional building model reconstruction based on detected shape and rooftop type. Besides the development of the methodology, the goal is to investigate the applicability and accuracy which can be accomplished in this context for different stereo sensor data. We use DSMs of Munich city which are obtained from different satellite (Cartosat-1, Ikonos, WorldView-2) and airborne sensors (3K camera, HRSC, and LIDAR). The paper later focuses on a quantitative comparisons of the outputs from the different multi-view sensors for a better understanding of qualities, capabilities and possibilities for applications. Results look very promising even for the DSMs derived from satellite data.