VARIATIONAL DISTRIBUTION DESIGNS FOR APPROXIMATE THOMPSON SAMPLING IN DEEP REINFORCEMENT LEARNING

Abstract

Exploration is a vital ingredient in reinforcement learning algorithms that has largely contributed to its success in various applications. Standard naive exploration strategies used in deep reinforcement learning are effective in simple tasks, but do not perform well in tasks with high dimensional state-action spaces as they are undirected. Thompson sampling is a directed, well-known and principled approach for balancing exploration and exploitation. But it requires the posterior distribution over the action-value functions or environment models to be maintained; this is generally computationally intractable for tasks that have a high dimensional state-action space. In this thesis, we argue that incorporating domain knowledge during the formulation of variational distributions for approximating these posterior distributions is useful in reinforcement learning. We explore this assertion by designing variational distributions, namely SANE and EVaDE for two different scenarios in the model-free and model-based reinforcement learning settings respectively.