Today's airborne SAR sensors provide geometric resolution in the order well below half a meter. Many features of urban objects become visible in such data. However, layover and occlusion issues inevitably arise in urban areas complicating automated object detection. In order to support interpretation, SAR data may be analyzed using complementary information from maps or optical imagery. In this paper, an approach for building detection in urban areas based on object features extracted from high-resolution interferometric SAR (InSAR) data and one orthophoto is presented. Features describing local evidence as well as context information are used. Buildings are detected by classification of those feature vectors within a Conditional Random Field (CRF) framework. Although as graphical model similar to Markov Random Fields (MRF), CRFs have the advantage of incorporating global context information, of relaxing the conditional independence assumption between features, and of a more general integration of observations. We show that, first, CRFs perform well in comparison to Maximum Likelihood classifiers and MRFs. Second, the combined use of optical and InSAR features may improve detection results.