Similarity detection among data files - a machine learning approach

Abstract

In any database, description files are essential to understand the data files in it. However, it is not uncommon that one is left with data files without any description file. An example is the aftermath of a system crash; other examples are related to security problems. Manual determination of the subject of a data file can be a difficult and tedious task particularly if files are look-alike. An example is a big survey database where data files that look alike are actually related to different subjects. Two data files on the same subject will probably have similar semantic structures of attributes. We detect the similarity between two attributes. Then we create clusters of attributes to compare the similarity of the subjects of two data files. And finally a machine learning technique is used to predict the subject of unseen data files.