Scalable Data Analysis on MapReduce-based Systems

Abstract

Many of today's applications, such as scientific, financial and social networking applications, are generating and collecting data at an alarming rate. As the size of data grows, it becomes increasingly challenging to analyze these datasets. The high computation and I/O cost of processing large amount of data make it difficult for these applications to meet the performance demands of end-users.

Meanwhile, the MapReduce framework has emerged as a powerful parallel computation paradigm for data processing on large-scale clusters. As such, there has been much effort in developing MapReduce-based algorithms to improve performance. However, there remain many challenges in exploiting MapReduce for efficient data analysis.

Thus, in this thesis, we develop new scalable, efficient and practical parallel data processing algorithms, frameworks and systems for computation intensive analysis and data intensive analysis on MapReduce-based systems. Specially, we explore two extremely important and challenging analyses: combinatorial statistical analysis and online analytical processing (OLAP) cube analysis.

The experimental results demonstrated the efficiency, effectiveness and scalability of our techniques. We believe that our research in this thesis brings us one step closer towards developing scalable and efficient big data analysis systems.