DebiNet: Debiasing Linear Models with Nonlinear Overparameterized Neural Networks

Recent years have witnessed strong empirical performance of\nover-parameterized neural networks on various tasks and many advances in the\ntheory, e.g. the universal approximation and provable convergence to global\nminimum. In this paper, we incorporate over-parameterized neural networks into\nsemi-parametric models to bridge the gap between inference and prediction,\nespecially in the high dimensional linear problem. By doing so, we can exploit\na wide class of networks to approximate the nuisance functions and to estimate\nthe parameters of interest consistently. Therefore, we may offer the best of\ntwo worlds: the universal approximation ability from neural networks and the\ninterpretability from classic ordinary linear model, leading to both valid\ninference and accurate prediction. We show the theoretical foundations that\nmake this possible and demonstrate with numerical experiments. Furthermore, we\npropose a framework, DebiNet, in which we plug-in arbitrary feature selection\nmethods to our semi-parametric neural network. DebiNet can debias the\nregularized estimators (e.g. Lasso) and perform well, in terms of the\npost-selection inference and the generalization error.\n