PMJoin: Optimizing distributed multi-way stream joins by stream partitioning

Abstract

In emerging data stream applications, data sources are typically distributed. Evaluating multi-join queries over streams from different sources may incur large communication cost. As queries run continuously, the precious bandwidths would be aggressively consumed without careful optimization of operator ordering and placement. In this paper, we focus on the optimization of continuous multi-join queries over distributed streams. We observe that by partitioning streams into sub-streams we can significantly reduce the communication cost and hence propose a novel partition-based join scheme - PM Join. A few partitioning techniques are studied. To generate the query plan for each substream, a heuristic algorithm is proposed based on a rate-based model. Results from an extensive experimental study show that our techniques can sufficiently reduce the communication cost. © Springer-Verlag Berlin Heidelberg 2006.