STOCHASTIC BREGMAN GRADIENT-TYPE ALGORITHMS FOR NONCONVEX OPTIMIZATION AND CONVEX-CONCAVE SADDLE POINT PROBLEMS

Abstract

We explore stochastic Bregman gradient methods for solving nonconvex stochastic optimization and convex-concave saddle point problems, crucial in statistics, image processing, and deep learning. Structured into three parts, we first consider nonconvex composite objective functions without globally Lipschitz continuous gradients for the differential part, introducing the stochastic Bregman proximal gradient (SBPG) method and its momentum-based version (MSBPG). MSBPG exhibits superior efficiency and robustness in training deep neural networks. The second part examines the long-term behavior of stochastic Bregman conservative field methods (SBCFM) in nonsmooth nonconvex problems, emphasizing their applicability to train the nonsmooth deep neural networks. The third part focuses on solving convex-concave stochastic saddle point problems beyond Lipschitz smoothness using the stochastic Bregman primal-dual method (SBPD). The method achieves sub-linear convergence rates and is enhanced with semi-stochastic variance reduction (S2BPD) for accelerated performance. Numerical results validate the efficiency of S2BPD in solving practical problems.