Development of intelligent network storage system with adaptive decision-making

Abstract

An enormous amount of digitization of content has led to extreme demands on storage systems. Multitier storage systems built with parallel distributed file-system enhance their scalability and I/O performance, while achieving cost-effectiveness in handling large data. However, these storage systems are not flexible enough to adapt to the complexity and dynamic environment of data. Designing a storage system to be more intelligent is a proven technique to overcome the challenges of managing complex data in a dynamic environment and hence, this paper proposes an Intelligent Network Storage System (INSS) with adaptive decision-making capability. An intelligent agent is introduced for autotiering and it can dynamically decide which storage tier data can be relocated based on their recent access patterns. The intelligent agent's design and implementation were inspired by the human brain and consequently, the functions from each of brain's five core regions were developed in INSS prototype. The intelligent agent utilizes Reinforcement Learning (RL) which guides the system itself to learn the optimal policies in relocating data. The prototype was built on a test bed with Lustre: A SAN file system for Linux with one metadata server and 3 storage nodes. With its adaptive and real-time decisionmaking capability, the policies optimized by RL were evaluated and the analysis has shown to achieve a high degree of accuracy with no performance penalty.