ZO-DARTS: Differentiable Architecture Search with Zeroth-Order Approximation

Neural Architecture Search (NAS) is a silver bullet in alleviating time consumption and human effort for deep neural network design. It is however challenging to search for good architectures with low consumption. In this paper, we propose a novel NAS framework to address the differentiable neural architecture search problem by inspecting the bi-level problem formulation from scratch. Combined with the Zeroth-Order (ZO) gradient descent technique and implicit gradients, the proposed algorithm can not only reduce search time for suitable architectures than existing works but maintain the final accuracy simultaneously. Experimental results show the efficacy of our proposed ZO-based NAS approach.