Please use this identifier to cite or link to this item:
|Title:||Parallelizing stateful operators in a distributed stream processing system: How, should you and how much?|
|Citation:||Wu, S.,Kumar, V.,Wu, K.-L.,Ooi, B.C. (2012). Parallelizing stateful operators in a distributed stream processing system: How, should you and how much?. Proceedings of the 6th ACM International Conference on Distributed Event-Based Systems, DEBS'12 : 278-289. ScholarBank@NUS Repository. https://doi.org/10.1145/2335484.2335515|
|Abstract:||We consider a distributed stream processing application, expressed as a data-flow graph with operators as vertices connected by streams and deployed over a cluster of compute nodes, where a small subset of the operators are often the performance bottlenecks for the entire application. In cases where a bottleneck operator is stateless, it is obvious that parallelization by splitting the incoming stream among multiple parallel operators deployed on different nodes can help improve performance. However, it is not so obvious when the bottleneck operator is stateful. In such a case, parallelization is much more challenging as it often requires a state sharing mechanism for the parallel operators. Moreover, it incurs additional overheads of required accesses by the parallel operators to shared state and synchronization constructs. In this paper, we propose a parallelization framework for stateful stream processing operators. The framework not only addresses issues related to the system model and support for operator parallelization, but also delves into the theoretical details that model the suitability of parallelization and the optimal degree of parallelism. We have implemented and evaluated our framework in the context of IBM's System S distributed stream processing middleware. While microbenchmarks are used to validate the proposed theoretical model, a parallelized implementation of a moving KNN application is used for the purpose of evaluation. Copyright © 2012 ACM.|
|Source Title:||Proceedings of the 6th ACM International Conference on Distributed Event-Based Systems, DEBS'12|
|Appears in Collections:||Staff Publications|
Show full item record
Files in This Item:
There are no files associated with this item.
checked on Jan 11, 2019
checked on Dec 29, 2018
Items in DSpace are protected by copyright, with all rights reserved, unless otherwise indicated.