Beyond RGB: A Real World Dataset for Multispectral Imaging in Mobile Devices

Multispectral (MS) imaging systems have a wide range of applications for computer vision and computational photography tasks, but do not yet enjoy widespread adoption due to their prohibitive costs. Recently, advances in the design and fabrication of photonic metamaterials have enabled the development of MS sensors suitable for integration into consumer grade mobile devices. Augmenting existing RGB cameras and their processing algorithms with richer spectral information has the potential to be utilized in many steps of the image processing pipeline, but diverse real world datasets suitable for conducting such research are not freely available. We introduce Beyond RGB
¹
, a real-world dataset comprising thousands of multispectral and RGB images in diverse real world and lab conditions that is suitable for the development and evaluation of algorithms utilizing multispectral and RGB data. All the scenes in our dataset include a colorimetric reference and a measurement of the spectrum of the scene illuminant. Additionally, we demonstrate the practical use of our dataset through the introduction of a novel illuminant spectral estimation (ISE) algorithm. Our algorithm surpasses the current state-of-the-art (SoTA) by up to 58.6% on established benchmarks and sets a baseline performance on our own dataset.