TOWARDS SCALABLE GRADIENT-BASED HYPERPARAMETER OPTIMIZATION IN DEEP NEURAL NETWORKS

Abstract

It is well-known that the performance of large-sized deep neural networks (DNNs) is sensitive to the setting of their hyperparameters. Hyperparameter optimization is thus recognized as a crucial step in the process of applying DNNs to achieve best performance and drive industrial applications. The works described in this thesis represent the first forays into the scalable gradient-based methods for elementary- and hyper-parameter optimization in DNNs in a unified manner.