Multi-agent systems on wireless sensor networks : a distributed reinforcement learning approach

Abstract

Implementing a multi-agent system (MAS) on a wireless sensor network comprising sensoractuatornodes is very promising as it has the potential to tackle the resource constraintsinherent in wireless sensor networks by efficiently coordinating the activities among thenodes. In fact, the processing and communication capabilities of sensor nodes enable themto make decisions and perform tasks in a coordinated manner in order to achieve somedesired system-wide or global objective that they could not achieve by their own.In this thesis, we review the research work about multi-agent learning and learning ofcoordination in cooperative MAS. We then study the behavior and performance of severaldistributed reinforcement learning (DRL) algorithms: (i) fully distributed Q-learningand its centralized counterpart, (ii) Global Reward DRL, (iii) Distributed Reward andDistributed Value Function, (iv) Optimistic DRL, (v) Frequency Maximum Q-learning(FMQ) that we have extended to multi-stage environments, (vi) Coordinated Q-Learningand (vii) WoLF-PHC. Furthermore, we have designed a general testbed in order to studythe problem of coordination in a MAS and to analyze more into detail the aforementionedDRL algorithms. We present our experience and results from simulation studies and actual implementation of these algorithms on Crossbow Mica2 motes, and compare their performancein terms of incurred communication and computational costs, energy consumptionand other application-level metrics. Issues such as convergence to local or global optima,as well as speed of convergence are also investigated. Finally, we discuss the trade-offsthat are necessary when employing DRL algorithms for coordinated decision-making tasksin wireless sensor networks when different level of resource-constraints are considered.