Assessing Sensitivity of Machine Learning Predictions.A Novel Toolbox\n with an Application to Financial Literacy

Despite their popularity, machine learning predictions are sensitive to\npotential unobserved predictors. This paper proposes a general algorithm that\nassesses how the omission of an unobserved variable with high explanatory power\ncould affect the predictions of the model. Moreover, the algorithm extends the\nusage of machine learning from pointwise predictions to inference and\nsensitivity analysis. In the application, we show how the framework can be\napplied to data with inherent uncertainty, such as students' scores in a\nstandardized assessment on financial literacy. First, using Bayesian Additive\nRegression Trees (BART), we predict students' financial literacy scores (FLS)\nfor a subgroup of students with missing FLS. Then, we assess the sensitivity of\npredictions by comparing the predictions and performance of models with and\nwithout a highly explanatory synthetic predictor. We find no significant\ndifference in the predictions and performances of the augmented (i.e., the\nmodel with the synthetic predictor) and original model. This evidence sheds a\nlight on the stability of the predictive model used in the application. The\nproposed methodology can be used, above and beyond our motivating empirical\nexample, in a wide range of machine learning applications in social and health\nsciences.\n