A Decoupled Multi-Task Network for Shadow Removal

Shadow removal, which aims to restore the illumination in shadow regions, is challenging due to the diversity of shadows in terms of location, intensity, shape, and size. Different from most multi-task methods, which design elaborate multi-branch or multi-stage structures for better shadow removal, we introduce feature decomposition to learn better feature representations. Specifically, we propose a single-stage and decoupled multi-task network (DMTN) to explicitly learn the decomposed features for shadow removal, shadow matte estimation, and shadow image reconstruction. First, we propose several coarse-to-fine semi-convolution (SMC) modules to capture features sufficient for joint learning of these three tasks. Second, we design a theoretically supported feature decoupling layer to explicitly decouple the learned features into shadow image features and shadow matte features via weight reassignment. Last, these features are converted to a target shadow-free image, affiliated shadow matte, and shadow image, supervised by multi-task joint loss functions. With multi-task collaboration, DMTN effectively recovers the illumination in shadow areas while ensuring the fidelity of non-shadow areas. Experimental results show that DMTN competes favorably with state-of-the-art multi-branch/multi-stage shadow removal methods, while maintaining the simplicity of single-stage methods.

I. Introduction

The shadow is a ubiquitous physical phenomenon in nature and is formed when direct illumination is blocked by an object. Shadows often degrade the performance of some computer vision tasks, such as segmentation, detection, recognition, and tracking [1], [2], [3], [4], [5], [6], [7], [8], [9], [10]. Shadow removal can be incorporated into these tasks to improve their robustness to direct illumination variations. Early approaches [11], [12], [13], [14], [15], [16], [17] determined illumination parameters to remove shadows by physically modeling them. These methods, however, highly rely on prior knowledge (such as illumination conditions and gradients [11], [18], [19]), and often work poorly in the umbra or penumbra regions.