Monitoring Network Data Streams

Abstract

The data input of a new class of applications such as network monitoring, web contents analysis and sensor networks takes the form of a stream, called data stream. This type of data is characterized by an extremely high data arrival rate and a very large data volume. Network monitoring may be the most compelling application that deals with data streams. The backbone of a large Internet service provider can generate 500 gigabytes data per day, even with a high degree of sampling. Data stream algorithms are essential to the efficient processing of such data. Major network tasks such as traffic management and security exploit two basic operations: aggregation and nearest neighbor search. This thesis addresses the problem of efficient processing of these two types of queries. We base our study on Gigascope, a real data stream management system deployed in AT&T, which is specially designed for processing network data streams.