Please use this identifier to cite or link to this item:
https://doi.org/10.2197/ipsjjip.vol.20.65
Title: | E3: An elastic execution engine for scalable data processing | Authors: | Chen, G. Chen, K. Jiang, D. Ooi, B.C. Shi, L. Vo, H.T. Wu, S. |
Keywords: | Cloud computing Elastic exection engine Parallel processing |
Issue Date: | 2012 | Citation: | Chen, G.,Chen, K.,Jiang, D.,Ooi, B.C.,Shi, L.,Vo, H.T.,Wu, S. (2012). E3: An elastic execution engine for scalable data processing. Journal of Information Processing 20 (1) : 65-76. ScholarBank@NUS Repository. https://doi.org/10.2197/ipsjjip.vol.20.65 | Abstract: | With the unprecedented growth of data generated by mankind nowadays, it has become critical to de- velop efficient techniques for processing these massive data sets. To tackle such challenges, analytical data processing systems must be extremely efficient, scalable, and flexible as well as economically effective. Recently, Hadoop, an open-source implementation of MapReduce, has gained interests as a promising big data processing system. Although Hadoop offers the desired flexibility and scalability, its performance has been noted to be suboptimal when it is used to process complex analytical tasks. This paper presents E3, an elastic and efficient execution engine for scalable data processing. E3 adopts a "middle" approach between MapReduce and Dryad in that E3 has a simpler communication model than Dryad yet it can support multi-stages job better than MapReduce. E3 avoids reprocessing intermediate results by adopting a stage-based evaluation strategy and collocating data and user-defined (map or reduce) functions into independent processing units for parallel execution. Furthermore, E3 supports block-level indexes, and built-in functions for specifying and optimizing data processing flows. Benchmarking on an in-house cluster shows that E3 achieves significantly better performance than Hadoop, or put it another way, building an elastically scalable and efficient data processing system is possible. © 2012 Information Processing Society of Japan. | Source Title: | Journal of Information Processing | URI: | http://scholarbank.nus.edu.sg/handle/10635/39475 | ISSN: | 03875806 | DOI: | 10.2197/ipsjjip.vol.20.65 |
Appears in Collections: | Staff Publications |
Show full item record
Files in This Item:
There are no files associated with this item.
Items in DSpace are protected by copyright, with all rights reserved, unless otherwise indicated.