EO space and multi-source data visualization using Virtual Reality in the ESA Φ-Lab

Since its inception in 2017, the European Space Agency’s Φ-Lab has put some of its focus on Virtual Reality (VR), Augmented Reality (AR) and Mixed Reality (MR), and how such technologies can be exploited in the frame of Earth Observation (EO) use-cases. Together with NORCE, the Norwegian Research Centre, a setup has been developed and deployed physically in the Φ-Lab in 2018, based on the GeoViz software, which allows navigating a full 3D globe of the Earth and even flying to other planets/satellites (e.g. the Moon). Users are free to choose which locations to explore and, for each of those locations, which EO images/products to examine (Sentinel-2 band combinations, Sentinel-3 scenes and Sentinel-1-derived products such as interferograms, coherence, intensity). GeoViz is also able to display 3D models (e.g. of spacecraft, buildings), point clouds (e.g. of cities, sites, inSAR showing soil subsidence), laser scans (e.g. of plantations) and even hyperspectral data. More recently, the hype around Digital Twins, and particularly the ESA Digital Twin Urban Pilot (DTUP) project, led to the generation of highly-detailed 3D model reconstructions of sites of interest (namely the ESA/ESRIN site and the city of Frascati), surveyed using drones. Using these 3D models partly as direct inputs, partly as inspiration, a second permanent VR/XR installation was developed and deployed in the lab in 2021. This installation leverages the Unity game engine development platform, to produce an early extended reality (XR) precursor of a digital twin of the ESRIN campus. The ESRIN 3D model is integrated, through the Unity application, with real-time environmental data sensed in-situ by two different kinds of specialized device existing in ESRIN, one measuring air quality (PM25, PM10, CO2, NO2, NH3, CO, etc.) and one measuring vegetation health (NDVI, evapotranspiration, chlorophyl, etc.). The complete augmented model can then be explored using virtual reality headsets. Future plans include the implementation of an Augmented Reality experience using an Android mobile phone App, which would allow users present on-site to walk around and use their phone cameras for the augmented visualization and interaction experience.