Probabilistic outlier detection for sparse multivariate geotechnical site investigation data using Bayesian learning

Abstract

Various uncertainties arising during acquisition process of geoscience data may result in anomalous data instances (i.e., outliers) that do not conform with the expected pattern of regular data instances. With sparse multivariate data obtained from geotechnical site investigation, it is impossible to identify outliers with certainty due to the distortion of statistics of geotechnical parameters caused by outliers and their associated statistical uncertainty resulted from data sparsity. This paper develops a probabilistic outlier detection method for sparse multivariate data obtained from geotechnical site investigation. The proposed approach quantifies the outlying probability of each data instance based on Mahalanobis distance and determines outliers as those data instances with outlying probabilities greater than 0.5. It tackles the distortion issue of statistics estimated from the dataset with outliers by a re-sampling technique and accounts, rationally, for the statistical uncertainty by Bayesian machine learning. Moreover, the proposed approach also suggests an exclusive method to determine outlying components of each outlier. The proposed approach is illustrated and verified using simulated and real-life dataset. It showed that the proposed approach properly identifies outliers among sparse multivariate data and their corresponding outlying components in a probabilistic manner. It can significantly reduce the masking effect (i.e., missing some actual outliers due to the distortion of statistics by the outliers and statistical uncertainty). It also found that outliers among sparse multivariate data instances affect significantly the construction of multivariate distribution of geotechnical parameters for uncertainty quantification. This emphasizes the necessity of data cleaning process (e.g., outlier detection) for uncertainty quantification based on geoscience data. © 2020 China University of Geosciences (Beijing) and Peking University