Please use this identifier to cite or link to this item: https://doi.org/10.1145/2335484.2335515
Title: Parallelizing stateful operators in a distributed stream processing system: How, should you and how much?
Authors: Wu, S.
Kumar, V.
Wu, K.-L.
Ooi, B.C. 
Keywords: Parallelization
Shared state
Stream processing
Issue Date: 2012
Source: Wu, S.,Kumar, V.,Wu, K.-L.,Ooi, B.C. (2012). Parallelizing stateful operators in a distributed stream processing system: How, should you and how much?. Proceedings of the 6th ACM International Conference on Distributed Event-Based Systems, DEBS'12 : 278-289. ScholarBank@NUS Repository. https://doi.org/10.1145/2335484.2335515
Abstract: We consider a distributed stream processing application, expressed as a data-flow graph with operators as vertices connected by streams and deployed over a cluster of compute nodes, where a small subset of the operators are often the performance bottlenecks for the entire application. In cases where a bottleneck operator is stateless, it is obvious that parallelization by splitting the incoming stream among multiple parallel operators deployed on different nodes can help improve performance. However, it is not so obvious when the bottleneck operator is stateful. In such a case, parallelization is much more challenging as it often requires a state sharing mechanism for the parallel operators. Moreover, it incurs additional overheads of required accesses by the parallel operators to shared state and synchronization constructs. In this paper, we propose a parallelization framework for stateful stream processing operators. The framework not only addresses issues related to the system model and support for operator parallelization, but also delves into the theoretical details that model the suitability of parallelization and the optimal degree of parallelism. We have implemented and evaluated our framework in the context of IBM's System S distributed stream processing middleware. While microbenchmarks are used to validate the proposed theoretical model, a parallelized implementation of a moving KNN application is used for the purpose of evaluation. Copyright © 2012 ACM.
Source Title: Proceedings of the 6th ACM International Conference on Distributed Event-Based Systems, DEBS'12
URI: http://scholarbank.nus.edu.sg/handle/10635/40614
ISBN: 9781450313155
DOI: 10.1145/2335484.2335515
Appears in Collections:Staff Publications

Show full item record
Files in This Item:
There are no files associated with this item.

SCOPUSTM   
Citations

12
checked on Dec 13, 2017

Page view(s)

51
checked on Dec 16, 2017

Google ScholarTM

Check

Altmetric


Items in DSpace are protected by copyright, with all rights reserved, unless otherwise indicated.