UNDERSTANDING AND IMPROVING NEURAL ARCHITECTURE SEARCH

Abstract

Despite recent advances in Neural Architecture Search (NAS), there are still certain essential aspects of NAS that have not been well investigated in the literature. Firstly, only a few efforts have been devoted to understanding the neural architectures selected by popular NAS algorithms in the literature. In the first work, we take the first step of investigating this problem. Secondly, standard NAS algorithms typically aim to select only a single architecture. So, in the second work, we present Neural Ensemble Search via Bayesian Sampling (NESBS) framework that can select better-performing neural ensembles. Thirdly, the search efficiency of NAS algorithms usually is limited by the need for model training. To this end, we propose NAS at Initialization (NASI) algorithm. Finally, the reason why training-free NAS using training-free metrics performs well remains a mystery in the literature and thus is studied in the last work.