Multi-Modal Transformer for Compressive LiDARs Using Hyperspectral Imaging Side-Information

Compressive satellite LiDAR (CS-LiDAR) has been recently introduced as a radically different computational sensing and reconstruction approach for LiDAR sensing of Earth. It is based on NASA’s adaptive wavelength scanning LiDAR (AWSL) system. Unlike conventional 1D LiDAR methods, CS-LiDAR utilizes sparse coded laser illumination across a 2D field-of-view. The aim is to compressively capture Earth from hundreds of kilometers above, enabling computational 3D imagery reconstruction with resolution that is comparable to that attained with data collected from just hundreds of meters. The forward imaging model captures the light propagation phenomena affecting the photon pulses transmitted from the sensor to the Earth’s surface and back. This work enhances CS-LiDAR by integrating imaging spectroscopy into a multimodal system and employing a transformer network for the inverse imaging problem, driven by multimodal attention mechanisms. Emulations enabled by enormous observational LiDAR data of Earth, available from NASA’s G-LiHT imaging observatory, highlight the efficacy of methods developed.