Towards an internet-scale stream processing service with loosely-coupled entities

Abstract

Stream processing engines are application-independent, specially designed query engines to process high-volume, real-time data streams. In recent year, many stream processing engines have been developed and employed by business entities to provide stream processing service over the internet. However, due to the inherent limitations of those stream processing engines, these entities suffer from scalability, over-investment and availability problems. A system incorporating those entities, promoting joint cooperation can achieve better system resource utilization, economical efficiency and scalability. In this thesis, we present the architecture of a scalable distributed stream processing system made up of loosely coupled entities. It provides two layers of services: The query layer service is capable of distributing queries to achieve load balance and minimize communication cost. The data layer service minimizes the communication cost of disseminating data streams from stream source nodes to entities. Our simulation studies show the efficiency of our proposed techniques.