Correlation-based methods for data cleaning, with application to biological databases

Abstract

Data cleaning aims at improving data quality through detecting and eliminating data artifacts that hamper the efficacy of analysis or data mining. Despite the importance, data cleaning remains neglected in certain knowledge-driven domains such as Bioinformatics. An in-depth study of real-world biological databases indicates that the biological data quality problem is multi-factorial and requires a number of different data cleaning approaches. Current data cleaning approaches that derive observations of data artifacts from the attribute values are inadequate. This thesis exploits the correlations patterns between attributes to provide additional information of the relationships embedded within a data set for data cleaning. We propose three novel correlation-based data cleaning methods to detect outliers and duplicates, and apply them to biological databases as proof-of-concepts. Experimental results show the effectiveness of these correlation-based data cleaning methods in detecting data artifacts that existing approaches fall short of addressing.