EXAMINING REPLICABILITY OF MACHINE LEARNING METHODS FOR VARIABLE IMPORTANCE AND PERFORMANCE MEASURES

Abstract

This thesis attempts to examine the replicability of machine learning methods in two different aspects. The first set of studies examined the replicability of variable importance using empirical and simulated data with linear interaction and square terms. Machine learning performed similarly to regression for linear and interaction on predictive accuracy and variable importance; for square data, machine learning methods performed better in both predictive accuracy and variable importance. However, for simulations on empirical data, this congruency was lost: better predictive accuracy did not lead to better replicability of variables. The second set of studies assessed the replicability of a common effect size of interest in machine learning methods, adapting replication procedures in psychology for these means. An empirical dataset was used to illustrate these methods; the results tentatively suggest that replication of explained variance is not guaranteed for machine learning, and different goals of replication can lead to divergent findings.