REGULARIZATION ON MACHINE LEARNING

Abstract

Deep neural networks have become a powerful tool for machine learning problems. However, overfitting frequently occurs. To achieve better generalization, many regularization methods were proposed to reduce overfitting. In this thesis, we propose a simple-yet-effective regularization method called Drop-Activation. At the training phase, we drop nonlinear activation functions randomly and set them to be identity functions. At the testing phase, a deterministic network with a new activation function is used and the new activation function is designed to average effect of the randomness of discarding activations. We theoretically deduce the implicit regularization terms of Drop-Activation and the effect of Drop-Activation can be considered as implicit parameter reduction. Also, our theoretical analysis verifies its capability to be used together with Batch Normalization (Ioffe and Szegedy 2015). We perform Drop-Activation on the benchmark datasets and show that the performance of popular networks can be improved generally by Drop-Activation.