IMG2DSM: Height Simulation From Single Imagery Using Conditional Generative Adversarial Net

This letter proposes a groundbreaking approach in the remote-sensing community to simulating the digital surface model (DSM) from a single optical image. This novel technique uses conditional generative adversarial networks whose architecture is based on an encoder-decoder network with skip connections (generator) and penalizing structures at the scale of image patches (discriminator). The network is trained on scenes where both the DSM and optical data are available to establish an image-to-DSM translation rule. The trained network is then utilized to simulate elevation information on target scenes where no corresponding elevation information exists. The capability of the approach is evaluated both visually (in terms of photographic interpretation) and quantitatively (in terms of reconstruction errors and classification accuracies) on subdecimeter spatial resolution data sets captured over Vaihingen, Potsdam, and Stockholm. The results confirm the promising performance of the proposed framework.

I. Introduction

Optical images are a valuable source of information for scene classification (semantic labeling) and object detection. In the investigation of such data, however, it is not possible to effectively differentiate objects composed of the same material (i.e., objects with the same spectral characteristics). For example, roofs and roads that are made of the same material exhibit the same spectral characteristics, which make the discrimination of such categories a laborious task using optical data alone. Conversely, elevation data [e.g., LiDAR and digital surface model (DSM)] provide rich height information but are unable to differentiate between objects with the same elevation that are made of different materials (e.g., roofs with the same elevation made of concrete or asphalt).