We present a framework for scene understanding from interferometric synthetic aperture radar data that is based on Bayesian machine learning and information extraction and fusion. A generic description of the data in terms of multiple models is automatically generated from the original signals. The obtained feature space is then mapped to user semantics representing urban scene elements in a supervised step. The procedure is applicable at multiple scales. We give examples of urban area classification and building recognition of Shuttle Radar Topography Mission data and of building reconstruction from submetric resolution Intermap data.