Sequential Gating Ensemble Network for Noise Robust Multiscale Face Restoration

Face restoration from low resolution and noise is important for applications of face analysis recognition. However, most existing face restoration models omit the multiple scale issues in the face restoration problem, which is still not well solved in the research area. In this paper, we propose a sequential gating ensemble network (SGEN) for a multiscale noise robust face restoration issue. To endow the network with multiscale representation ability, we first employ the principle of ensemble learning for SGEN network architecture design. The SGEN aggregates multilevel base-encoders and base-decoders into the network, which enables the network to contain multiple scales of receptive field. Instead of combining these base-en/decoders directly with nonsequential operations, the SGEN takes base-en/decoders from different levels as sequential data. Specifically, it is visualized that SGEN learns to sequentially extract high-level information from base-encoders in a bottom-up manner and restore low-level information from base-decoders in a top-down manner. Besides, we propose realizing bottom-up and top-down information combination and selection with a sequential gating unit (SGU). The SGU sequentially takes information from two different levels as inputs and decides the output based on one active input. Experimental results on the benchmark dataset demonstrate that our SGEN is more effective at multiscale human face restoration with more image details and less noise than state-of-the-art image restoration models. Further utilizing an adversarial training scheme, SGEN also produces more visually preferred results than other models under subjective evaluation.