Monte Carlo Tree Search and Delay-Aware Feedback Adaptation for Underwater Acoustic Link Tuning

Abstract

The unique properties of Underwater Acoustic Communication (UAC) channels such as limited bandwidth, strong multipath propagation, and large delay spread over tens or even hundreds of milliseconds result in severe non-stationary fading statistics. When the channel statistics change, the performance of a modulation scheme designed for a specific channel model might deteriorate. This motivates the need for real-time link tuning. We use adaptive modulation with a high degree of freedom in modulation and coding schemes to optimize channel throughput in the time-varying UAC channel. The key idea involves dealing with the exploration versus exploitation dilemma, which is formulated as a Markov Decision Process, to maximize average data rate without any prior knowledge of the UAC channel. Reinforcement learning methods help estimate Channel State Information and schedule packet transmissions to achieve higher throughput. We present a hybrid algorithm that includes short-term planning in MDPs to select MCSs. This helps us reduce computational complexity while achieving a comparable performance to the algorithms that perform full-planning. We also study an online learning strategy to determine an appropriate number of transmission packets between every two feedback packets to reduce the time spent on obtaining feedback, as the long propagation delay characteristic makes frequent feedback impractical.