EDGE OF CHAOS IN DEEP LEARNING MODELS AND ITS APPLICATION TO TRAINING ALGORITHMS

Abstract

Over the past decade, deep learning models have achieved remarkable success in many different practical areas. However, there is still no general framework to explain deep learning models. In this thesis, we present a framework to understand and improve deep neural networks with the edge of chaos. First, we demonstrate a three-phase behavior in deep neural networks and identify the exact edge of chaos as the boundary between the chaotic phase and the (pseudo)periodic cycle phase. Our experiments on various deep neural network models demonstrate that deep neural networks are optimal near the edge of chaos. Next, we study the role of various hyperparameters in modern neural network training algorithms in terms of the order-chaos phase diagram, and show how to improve the performance of neural networks with the edge of chaos. Lastly, we discuss several limitations of this framework as our future research directions.