Exploring the Intersection Between Neural Architecture Search and Continual Learning

Despite the significant advances achieved in deep learning, the deep neural networks’ (DNNs) design approach remains notoriously tedious, depending primarily on intuition, experience, and trial and error. This human-dependent process is often time-consuming and prone to errors. Furthermore, the models are generally bound to their training contexts, with no considerations to their surrounding environments. Continual adaptiveness and automation of neural networks is of paramount importance to several domains where model accessibility is limited after deployment (e.g., IoT devices, self-driving vehicles, etc.). Additionally, even accessible models require frequent maintenance postdeployment to overcome issues such as data/concept drift, which can be cumbersome and restrictive. By leveraging and combining approaches from neural architecture search (NAS) and continual learning (CL), more robust and adaptive agents can be developed. This study conducts the first extensive review on the intersection between NAS and CL, formalizing the prospective paradigm and outlining research directions for lifelong autonomous DNNs.