Jointly Learning Spectral Sensitivity Functions and Demosaicking via Deep Networks

Demosaicking is reconstructing a RGB image from the mosaicked image recorded by a single sensor with a color filter array (CFA), which has achieved great progress via deep learning. Some works on joint CFA design and demosaicking via deep learning have been presented. However, almost all existing approaches focus only on optimizing filter arrangements without considering the used spectral sensitivity functions (SSFs). The commonly used SSFs are to mimic human trichromatic perception, which are not optimal for deep-learning-based de-mosaicking. In this paper, we simultaneously learn the CFA and demosaicking using a deep convolutional neural network, where both the filter arrangement and used SSFs are optimized. By modeling the applying of SSFs as a convolutional layer with physical constraints, we formulate our joint learning approach as an end-to-end autoencoder, which can be trained as the standard convolutional neural networks. Our approach allows the designing of CFA patterns with arbitrary sizes, where the filter arrangements can be predefined or not. We demonstrate the effectiveness and advantages of our approach by comparing with fixed Gaussian functions and various predefined filter arrangements.