Rethinking the design of distributed stream processing systems

Abstract

In this paper, we present a novel architecture to support large scale stream processing services in a widely distributed environment. The proposed system, COSMOS, distinguishes itself by its loose coupling and communication efficiency. To exploit the sharing of data transfer incurred by different queries and to break the tight coupling of the distributed nodes, a new communication paradigm, content-based network, is employed. We discuss the design and the challenges of this system. © 2008 IEEE.