LightMix: Multi-Objective Search for Lightweight Mixed-Scale Convolutional Neural Networks

Lightweight convolutional neural network (CNN) design has been a research focus for recent years, justified by the popularity of deploying deep models on resource-constrained devices. Notably, depthwise separable (DW-Sep) convolutions have been extensively adopted in constructing CNNs to reduce computational complexity. However, the improved efficiency of DW-Sep convolutions usually brings a degradation in model expressivity/performance. To overcome this issue, this paper presents a lightweight mixed-scale convolution block, dubbed LightMix, for improving the model representation capability via multi-scale feature extraction and fusion while reducing the computational complexity through channel divisions. A multi-objective neural architecture search framework integrating the LightMix block is developed to automate the LightMix-based CNN architecture design. Furthermore, we propose a population grouping strategy to balance the difficulty between optimizing predictive accuracy and model complexity during the search process. This strategy is capable of retaining potentially promising architectures, contributing to an enhanced population diversity. The proposed method only takes 0.02 GPU days to discover excellent architectures that achieve 2.52% test error with 1.78 M parameters, and 17.19% test error with 1.81 M parameters on CIFAR-10 and CIFAR-100, respectively. On ImageNet, the architecture searched with 0.3 GPU days achieves a 22.1% Top-1 error rate with merely 4.8 M parameters and 488 M MAdds.