Summary
This paper considers the predictive problem setting in which the input features are dichotomized into two groups: explanatory features and contextual features. For such a setting, the authors propose the contextual lasso, a statistical estimator that leverages the expressiveness of deep neural networks to learn interpretable sparse linear models with sparsity patterns that can vary based on the contextual features. More precisely, the authors proposed using a neural network to model the nonlinear effects of the contextual features on the explanatory features. In addition, to enforce sparsity among the coefficients for the explanatory features, the network is trained with a regularizer in the form of a projection layer that maps the output (dense) to sparse coefficients by performing a projection onto the $\ell_1$-ball. Experiments on real and synthetic data suggest that the contextual lasso leads to models which remain interpretable and can be sparser than the regular lasso, without sacrificing the predictive accuracy of standard deep neural networks.
Strengths
This work appears to be the first to propose a sparsity-inducing projection layer that maps a dense output of a neural network onto a space of $\ell_1$-constrained linear models by solving a constrained quadratic program.
The benefit from the proposed estimator is that it can utilize the expressive power of deep neural networks and yet learn interpretable sparse linear models by encouraging contextual sparsity through an additional projection layer; all while allowing feature relevance to depend on the prediction context as opposed to determining a fixed set of relevant features. In addition to the classical contextual sparse learning setting, a variant of the sparsity-encouraging projection layer is also devised for the group lasso setting.
The proposed estimator and the formulation of the quadratic program for determining the sparse coefficients are technically sound and well-supported. Moreover, the particular sparse learning setting that is being considered is well-supported by examples where the input features are inherently dichotomized into explanatory features and contextual features, and the need for contextual sparsity in such a setting is well-justified.
The effectiveness of the contextual lasso has been evaluated on both synthetic and real data (in the standard setting and a setting with grouped explanatory features). The findings seem to suggest that, in contrast to other sparse linear models and a conventional DNN, which use either all or almost all available explanatory features, contextual lasso leads to linear models that are much simpler and involve less explanatory features on average.
The paper is rather well written and organized. The notation is very clear, easy to follow, unambiguous and consistent throughout the paper.
Weaknesses
The problem considered in this work is a rather general one, as many applications involve a certain context that can be leveraged to select candidate features and determine their effects. Nevertheless, this generality of the problem is not matched by the extensiveness of the conducted experiments. It seems that the contribution’s potential significance is limited by (1) the narrow settings of the experimental design (particularly in the synthetic data generation process) and (2) the consideration of only two real-world applications given the pervasiveness of sparse learning problems surrounded by a certain prediction context. I would encourage the authors to consider additional datasets characterized by both explanatory and contextual features.
Although this work is aimed at capturing the prediction context by leveraging the expressive power of neural networks to map contextual features to a sparse coefficient vector, there are other works that take different approaches to the same end. For instance, [1] proposed adaptive optimizers for structured sparsity for deep learning, but also discusses unstructured sparsity ($\ell_1$) which is directly relevant to this work. There are also other works, such as [2], that take a non-neural based approach to capturing prediction context but I believe are still relevant. Namely, [2] addresses the house pricing problem (provided as an example in Section 1 of this work) using a generalization of the group lasso to a graph setting, where the graph represents the context defined by the distances between the house locations. Finally, there are also approaches that are not directly applicable to the setting considered in this paper, however, in a more general sense, they are relevant to inducing sparsity in neural networks and should be considered as a part of the related work (e.g., [3]).
[1] Deleu, T., & Bengio, Y. (2021). Structured sparsity inducing adaptive optimizers for deep learning. arXiv preprint arXiv:2102.03869.
[2] Hallac, D., Leskovec, J., & Boyd, S. (2015, August). Network lasso: Clustering and optimization in large graphs. In Proceedings of the 21th ACM SIGKDD international conference on knowledge discovery and data mining (pp. 387-396).
[3] Wang, P., He, X., Li, G., Zhao, T., & Cheng, J. (2020, April). Sparsity-inducing binarized neural networks. In Proceedings of the AAAI conference on artificial intelligence (Vol. 34, No. 07, pp. 12192-12199).
As previously suggested, other sparse linear estimator variants have been proposed to utilize contextual information for the same or similar setting as the one considered in this work. Such models include Network Lasso (mentioned in the previous comment), Graphical Lasso and its associated extensions, among others. In that regard, I am wondering why the authors did not consider including such estimators as additional baselines. Moreover, since this work proposes a separate projection layer to induce sparsity in the output of a neural network, I also believe that a comparison to a conventional neural network with a dropout layer applied on its outputs ($\eta$ values) would be appropriate.
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Limitations
A separate “Limitations” section has not been included in this paper. No potential negative societal impact has been declared in the main manuscript. For my improvement suggestions, please refer to the “Weaknesses” section of this review.