Big Data Processing with Peer-To-Peer Architectures

Abstract

Recent developments have brought about the introduction of, what some may classify as, disruptive technologies into the peripheral of both researchers and developers alike; we frequently hear the mention of the Big Data paradigm, the deployment with cloud computing, the NoSQL movement, or the MapReduce framework. While some may have their reservations on these new concepts, their continual widespread adoption in the industry undoubtedly indicates previously unsatisfied needs for certain systemic providence from the solutions of yesteryear. Three such desirable qualities of system architecture can be identified: massive horizontal scalability, elastic resource consumption and robust distributed processing.

Currently, the predominant architecture adopted for modern data processing system is that of the master/workers architecture; this is said to be for the simplicity of the system design. However, it is perhaps profitable to investigate more elaborated alternatives, especially if systemic qualities may thus be enhanced. Extrapolating from the desirables, it appears that structured peer-to-peer (P2P) overlays present as a good match to the said conditions. This thesis sets out to demonstrate the feasibility of adopting a structured P2P overlay in the design of modern data processing system such that some of the identified systemic qualities may be magnified.