Efficient Failure Recovery in Large-scale Graph Processing Systems

Abstract

Wide range of applications in Machine Learning and Data Mining (MLDM) area have increasing demand on utilizing distributed environment to solve certain problems. It naturally results in the urgent requirements on how to make sure the reliability of large-scale graph processing systems. Traditional rollback recovery mechanisms in distributed system have been studied in various forms by a wide range of researchers, but not many of them are actually applied in real systems.

To address these challenges, in this thesis, we propose and evaluate two new rollback recovery algorithms specially designed for large-scale graph processing systems, called Log-Based Recovery and Shadow-Based Recovery, which aim at reducing the recovery time without introducing too much overhead. Compared to previous work, our new algorithms can achieve efficient failure recovery with quite good scalability. Our experimental evaluation shows that one of our algorithms, Shadow-Based Recovery, performs well in terms of both overhead and recovery time.