Enhancing Neural Architecture Search by Upgrading Weak Components

Neural Architecture Search (NAS) becomes increasingly popular due to the ability to liberate the researchers from the complicated model design. However, typical NAS methods search for high-quality networks at a substantial cost of time and resources. This work enhances the neural architecture search by upgrading the weak components of promising candidate networks. First, we define a search space stacked with repeated components, such as cells and blocks. Second, we propose an attention-wise component evaluation metric to score the components. Third, we exploit a novel search policy to discover weak and strong components in promising candidate networks. Lastly, we use strong components to replace weak components. Besides, to achieve higher accuracy with less time cost, we integrate two optimization schemes: knowledge transfer and self-ensemble. The knowledge transfer accelerates model training by integrating an attentive regularizer into the objective goal. And the self-ensemble method ensembles promising candidate networks as the teacher network to teach the student network. The experiment results demonstrate our method can achieve state-of-the-art results(97.48% accuracy on CIFAR-10 and 84.38% accuracy on CIFAR-100).