Even though neural networks have been long deployed in applications involving tabular data, still existing neural architectures are not explainable by design. In this paper, we propose a new class of interpretable neural networks for tabular data that are both deep and linear at the same time (i.e. mesomorphic). We optimize deep hypernetworks to generate explainable linear models on a per-instance basis. As a result, our models retain the accuracy of black-box deep networks while offering free-lunch explainability for tabular data by design. Through extensive experiments, we demonstrate that our explainable deep networks have comparable performance to state-of-the-art classifiers on tabular data and outperform current existing methods that are explainable by design.
Paper
Similar papers
Peer review
Summary
This work explores interpretable models with a focus on tabular data and introduces a novel method that is locally linear while retaining a non-linear global decision boundary. The authors achieve this by expressing their model as a linear model that takes the input features and produces a dot product with weights that depend non-linearly on the features. Thus for every point, a linear decision surface is produced but the surface remains parameterized as a non-linear function of the inputs. This makes the method more interpretable as feature importance attribution can be performed, while at the same time preserving the accuracy benefits. The authors perform a very broad evaluation of their method on the AutoML benchmark, showing that there method indeed outperforms white box models such as logistic regression, remaining on par with state-of-the art models for tabular data, while offering better and more efficient interpretability.
Strengths
1. The evaluation that the authors perform is very thorough and the results seem impressive. The formulated hypotheses are very sensible and very convincingly demonstrated. 2. The method is very intuitive and easy to understand. The toy task with the half moons is a really good illustration to gain intuition! The paper is also very well written and easy to follow.
Weaknesses
1. Input-dependent weights reminds me lot of a (single layer) Transformer, where the attention matrix (which is built from the inputs and weights in a non-linear fashion) serves as a weight matrix to then linearly combine the inputs. The method here uses more involved networks to produce this set of weights (not just a single layer) but still I feel like this similarity is really worth highlighting more. How would for instance a single-layer Transformer perform? Would the attention matrix also serve as a useful interpretability tool? The baseline TabNet seems to leverage attention scores to some degree to offer interpretability, it would be nice if the authors could further comment on the similarities. 2. While the method is linear by design around a fixed datapoint x, the decision surface could still rapidly change as a function of x (especially if the classification task requires the function to do so). In such cases, does the method still offer high interpretability? The feature importance metric proposed here only zeroes out the input feature applied to the hyperplane, but the input feature used for constructing the weights is not modified. In such a situation, I would expect that the feature importance metric would fail. I guess this is why the authors employ a regularizer in the loss? How does changing its strength affect the results? In general it would also be nice to get an understanding of where this method might fail to offer explainable results.
Questions
1. The authors focus on tabular data but in principle I don’t see anything stopping the methodology from also working on vision. Did the authors try any experiments on simple vision datasets? I would be very interested in seeing if the method can also keep high accuracy scores there. 2. How does the method compare to a simple first order Taylor expansion of the black-box model of interest, around a given input datapoint x? This strategy should also preserve global accuracy scores while offering interpretability locally. It is of course more expensive to compute compared to the authors model, but ignoring this, I would like to understand the differences better.
Rating
7
Confidence
3
Soundness
3
Presentation
3
Contribution
3
Limitations
See above.
Summary
The paper presents a hypernetwork approach to build an interpretable neural network for tabular data. Deep hypernetwork takes an input and returrns the weights for a linear model, which classify a given point. The example-baed interpretability follows from the interpretability of a linear models. The model is evaluated on typical benchmark dataset.
Strengths
1. The model is simple, very intuitive and theoretically sound. 2. This is another work which show a wide range of applicability of hypernetworks. Here, the authors use hypernetworks for interpretability problem. 3. The authors propose local classification rule (individual model for each point) using linear model (target network), while the global rule is given by the deep model (hypernetwork). This is a very interesting idea and has not been invistigated before for hypernetworks
Weaknesses
1. The authors restrict their baseline to interpretable classifiers. There are a lot of recent studies on tabular data that authors should compare with. 2. This is not the first work on uisng hypernetworks on tabular data. The authors of [1] use hypernetworks to build an ensemble of neural networks. The auhtors should at least refer to this work in their paper [1] Wydmański, Witold, Oleksii Bulenok, and Marek Śmieja. "Hypertab: Hypernetwork approach for deep learning on small tabular datasets." 2023 IEEE 10th International Conference on Data Science and Advanced Analytics (DSAA). IEEE, 2023.
Questions
-The authors should explain the connection with the paper [1] and use other deep models for comparison. -The global classification rule is given by the hypernetwork. How the classification changes for nearby points. The authors could illustrate it with an example.
Rating
7
Confidence
4
Soundness
3
Presentation
3
Contribution
3
Limitations
yes
Summary
This paper introduces a new neural network architecture called Interpretable Mesomorphic Networks (IMN) for handling tabular data. IMN combines the high accuracy of deep learning models with the interpretability of linear models by generating instance-specific linear models through deep hypernetworks, achieving both local linearity and global depth.
Strengths
S1: Innovative Design: IMN combines the advantages of deep learning and linear models, achieving a balance between high performance and interoperability. S2: Extensive Experimental Evidence: IMN's effectiveness and advantages are demonstrated through experiments on various datasets. S3: Interpretability: IMN provides both instance-level explanations and global feature importance analysis. S4: Efficiency: IMN shows excellent inference time performance, particularly on large-scale datasets.
Weaknesses
W1: The implementation and training process of IMN is relatively complex, which might pose challenges for deployment and maintenance in practical applications. W2: IMN relies on the design and optimization of deep hypernetworks, potentially requiring extensive hyperparameter tuning and computational resources. W3: While extensive, the experiments do not cover all possible application scenarios, especially for more challenging unstructured data. what is the significant of the proposed method?
Questions
Q1: How scalable is IMN in practical applications (i.e., in time complexity)? While IMN performs well in experiments, how does it fare in large-scale, complex tabular datasets in real-world applications? Q2: How sensitive is IMN to hyperparameters? Does IMN's performance vary significantly with different hyperparameter settings, requiring extensive tuning? Q3: Comparison with other emerging interpretability methods: How does IMN compare with the latest interpretability methods, such as those based on attention mechanisms, in terms of advantages and disadvantages?
Rating
6
Confidence
4
Soundness
2
Presentation
2
Contribution
2
Limitations
check weakness and questions.
References
**References**: [1] Kadra, A., Lindauer, M., Hutter, F., & Grabocka, J. (2021). Well-tuned simple nets excel on tabular datasets. Advances in neural information processing systems, 34, 23928-23941. [2] Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., ... & Duchesnay, É. (2011). Scikit-learn: Machine learning in Python. the Journal of machine Learning research, 12, 2825-2830. [3] Van Breugel, B. & Van Der Schaar, M.. (2024). Position: Why Tabular Foundation Models Should Be a Research Priority. Proceedings of the 41st International Conference on Machine Learning in Proceedings of Machine Learning Research. [4] Gijsbers, P., LeDell, E., Thomas, J., Poirier, S., Bischl, B., & Vanschoren, J. (2019). An open source AutoML benchmark. [5] McElfresh, D., Khandagale, S., Valverde, J., Prasad C, V., Ramakrishnan, G., Goldblum, M., & White, C. (2024). When do neural nets outperform boosted trees on tabular data?. Advances in Neural Information Processing Systems, 36.
Response to authors' reply
I appreciate your thorough response to my comments. While your clarifications address some of the concerns, I still have reservations about the interpretability study and the overall experimental design. **Interpretability study limitations** Your interpretability study (hypothesis 4) is based on a relatively small dataset (Census with only 10+ features) without noise data. The additional study on mushroom edibility prediction also uses a well-studied dataset where high performance (F1 score of 0.99) is easily achievable. These choices in experimental datasets limit the generalizability of your interpretability claims. **Real-world application cases** The paper would be significantly strengthened by including detailed case studies from real-world applications (not just well-studied datasets). These could demonstrate IMN's effectiveness and interpretability in practical scenarios such as financial risk assessment, healthcare predictions, marketing analytics, or HR decision support. **Regarding your dataset selection** To increase confidence in IMN's capabilities, it would be valuable to see results on a wider range of datasets, especially those with more complex feature interactions, higher dimensionality, and the presence of noise. While I acknowledge the positive scores from other reviewers, these remaining concerns prevent me from increasing my score.
Additional Response to Reviewer
We thank the reviewer for the valuable response. - Regarding; **"Dataset selection: To increase confidence in IMN's capabilities, it would be valuable to see results on a wider range of datasets, especially those with more complex feature interactions, higher dimensionality, and the presence of noise."** We agree that a wide range of datasets is necessary to showcase the gain of our method. For this reason, we already used a large battery of datasets in our experimental protocol. Concretely: - For Hypothesis 1 and 2 on performance vs. interpretable methods by design: - We compared against 7 baselines on 35 datasets from the popular AutoMLBenchmark [1], details in Table 6, Appendix C. - For Hypothesis 3 on local interpretability methods: - We compared against 8 baselines on the 3 datasets offered by the recent XAI Benchmark [2], details in Table 3, Section 5. - For Hypothesis 4 on global interpretability: - We compared against 4 baselines on 2 datasets from prior work [3, 4]. Overall, we compared against 14 different baselines on 40 different datasets across the four sets of experiments. The datasets in our protocol arise in various domains, such as healthcare (dataset: blood-transfusion), finance (dataset: census, australian, credit-g), etc. - Regarding **"Your interpretability study (hypothesis 4) is based on a relatively small dataset (Census with only 10+ features) without noise data. The additional study on mushroom edibility prediction also uses a well-studied dataset where high performance (F1 score of 0.99) is easily achievable. These choices in experimental datasets limit the generalizability of your interpretability claims."**: We would like to stress that the primary contribution of our work is offering a local interpretability technique that is by design interpretable due to the linear models generated by our hypernetworks. As described in the manuscript, the purpose of Hypothesis 4 is to show that even though our method is designed for local interpretability, it also offers global interpretability as a bonus feature. We would like to kindly point out to the reviewer that the census and mushroom-edibility datasets are well-known datasets from prior work [3, 4]. Regarding the well-studied problem of mushroom-edibility, we would kindly like to point out to the reviewer that we verify that our method provides valid global interpretability in the following ways: - The experimental way as provided for the Census dataset. Where a feature is dropped and the impact on the predictive performance of the model is measured as importance. - From knowing the actual feature importances. As such, for this second scenario, one needs a well-studied problem to know the underlying feature importance. Additionally, please note that the problem of feature attribution (i.e. finding important/influential features) is orthogonal to the performance of classifiers on a dataset. If a baseline achieves a high F1 score, it does not mean that the dataset is uninteresting from an interpretability perspective. - Regarding: **"The paper would be significantly strengthened by including detailed case studies from real-world applications (not just well-studied datasets). These could demonstrate IMN's effectiveness and interpretability in practical scenarios such as financial risk assessment, healthcare predictions, marketing analytics, or HR decision support."** As mentioned in our previous reply, we use datasets from diverse domains (see comment above on the datasets used). We would like to kindly point out to the reviewer that both census (from the 1994 Census database) and mushroom-edibility (from the National Audubon Society Field Guide) are real-world datasets as mentioned in [3]. Lastly, we would like to mention a few datasets that fall under the categories the reviewer lists: 1) blood-transfusion (Blood Transfusion Service Center in Hsin-Chu City Taiwan), 2) credit-g (German credit data), 3) australian (Australian Credit Approval dataset), etc. **We believe we have clarified all the concerns raised by the reviewer. We would kindly ask the reviewer to increase the score and recommend acceptance based on the provided clarifications.** [1] Gijsbers, P., LeDell, E., Thomas, J., Poirier, S., Bischl, B., & Vanschoren, J. (2019). An open source AutoML benchmark. [2] Liu, Y., Khandagale, S., White, C., & Neiswanger, W. Synthetic Benchmarks for Scientific Research in Explainable Machine Learning. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2). [3] Arik, S. Ö., & Pfister, T. (2021, May). Tabnet: Attentive interpretable tabular learning. In Proceedings of the AAAI conference on artificial intelligence (Vol. 35, No. 8, pp. 6679-6687). [4] Lundberg, S. M., & Lee, S. I. (2017). A unified approach to interpreting model predictions. Advances in neural information processing systems, 30.
Regarding the data selection
Dear authors, Could you provide the interpretability study (Hypo 4) on more (little bit larger) dataset, for instance, openml 618 and openml 579? Best
Reviewer Response
We would like to thank the reviewer for the provided reference. Regarding: **"Could you provide the interpretability study (Hypo 4) on more (little bit larger) dataset"** We would like to point out to the reviewer that the census dataset (id 1590) has 49k instances, while mushroom edibility dataset (id 24) has 1.8k instances. Datasets 579 and 618 referenced by the reviewer have 250 instances and 1k instances, so in this regard, they are not larger. Additionally, both datasets are synthetic datasets and not real-world examples compared to the original set considered. Despite the above, we gladly run on the datasets referenced by the reviewer. Both datasets are generated by the Friedman function + noise, given as: $f(x_1, x_2, x_3, x_4, x_5) = 10\sin(\pi x_1 x_2) + 20(x_3 - 0.5)^2 + 10x_4 + 5x_5 + noise$ Dataset 579 features 250 examples and the 5 original features. Dataset 618 is generated by the same function, however, it additionally has 45 random features, which is 9 times the original number of features. The dataset features 1000 instances. For both datasets, the first 5 features are the original features that contribute to the output. - **We run our method on dataset 579, and we get the following results:** Feature ranking: ['x_4', 'x_5', 'x_2', 'x_3', 'x_1'] Feature impacts: [0.7252574, 0.48875564, 0.4850915, 0.44188046, 0.3727691] Repeating the experiment as for the census dataset in Hypothesis 4, we drop a feature and evaluate the increase in error prediction by the model. We present the results in the Table below: |Feature to remove | Error change | |:--------------:|:------------:| |$x_4$| + 0.09453| |$x_5$| + 0.08712| |$x_2$| + 0.04188| |$x_3$| + 0.01331| |$x_1$| + 0.00539| The error change confirms the ranking generated by our method and is intuitive given the form of the equation. We would like to mention that the interpretability provided in this scenario is dependent on the dataset. As in this scenario, we do not have a Ground truth *ranking* of feature importances. - **We run our method on dataset 618, and we get the following results:** Feature ranking: ['x_5', 'x_4', 'x_3', 'x_2', 'x_1', 'x_41', 'x_21', 'x_19', 'x_38', 'x_50', 'x_6', 'x_28', 'x_17', 'x_44', 'x_35', 'x_22', 'x_24', 'x_48', 'x_14', 'x_29', 'x_15', 'x_11', 'x_36', 'x_26', 'x_42', 'x_32', 'x_46', 'x_23', 'x_43', 'x_12', 'x_16', 'x_39', 'x_33', 'x_45', 'x_34', 'x_30', 'x_18', 'x_27', 'x_40', 'x_10', 'x_7', 'x_49', 'x_13', 'x_8', 'x_25', 'x_47', 'x_20', 'x_37', 'x_9', 'x_31'] Feature impacts: [0.7121832 0.3526064 0.3483662 0.23299913 0.15339686 0.09143675 0.07042366 0.05917934 0.05624102 0.0553472 0.0540031 0.05357284 0.05347642 0.05335978 0.05294265 0.05056856 0.05014918 0.04853891 0.04850458 0.04743053 0.04738801 0.04495728 0.04370673 0.04227468 0.04207935 0.0419944 0.04045379 0.03908753 0.03472474 0.03439515 0.03094473 0.03029181 0.02994934 0.02873245 0.02851641 0.02840919 0.02693252 0.02679885 0.02547741 0.02534737 0.02332983 0.02322569 0.02239586 0.02127747 0.02053387 0.02039642 0.02035189 0.01959292 0.01916017 0.01906535] In this scenario, there exists a Ground truth ranking, since we have features that contribute to the target variable and features that are random. Based on the above results, we conclude that all the non-noise variables are captured by IMN as important even in a highly noisy scenario, where 90% of the features are random and only 10% contribute to the target variable.
Dear authors, > **larger** means including more features. I think this experiment on the generated dataset is interesting. (Could you provide more details on this Exp and other baseline methods?) My questions on experiments are all addressed. I will raise my score accordingly.
Summary
The paper introduces a new class of neural networks designed to be both deep and interpretable. These networks, referred to as Interpretable Mesomorphic Networks (IMN), utilize deep hypernetworks to generate linear models on a per-instance basis. This approach aims to retain the high accuracy of traditional black-box neural networks while providing explainability by design. The authors demonstrate through extensive experiments that their models perform comparably to state-of-the-art classifiers and outperform existing explainable methods on tabular data.
Strengths
- The proposed method offers interpretability by generating linear models specific to each data instance, making it easier to understand individual predictions. - The IMN achieves accuracy comparable to black-box models, ensuring high performance while providing explainability. - The approach is applicable to both classification and regression tasks, enhancing its utility across different types of tabular data problems. - The method has been validated through extensive experiments, demonstrating its efficacy compared to state-of-the-art classifiers and explainable methods.
Weaknesses
I am not an expert on interpretability issues, but I am fairly familiar with the topic of tabular data prediction. Overall, I didn't notice any significant shortcomings in the paper. However, the author overlooked DANet `[1]`, a model that outperforms TabNet and provides a better interpretable framework. This could be included for comparison. `[1]` Danets: Deep abstract networks for tabular data classification and regression
Questions
See above
Rating
6
Confidence
2
Soundness
3
Presentation
3
Contribution
3
Limitations
N/A
Good work
I would like to thank the authors their answer and I hope to see the paper at NeurIPS!
Reviewer Reply
We thank the reviewer for reading our rebuttal and increasing the score from 3 to 4. However, we addressed the reviewer's concerns thoroughly in our rebuttal. In particular, we explained that the three weaknesses mentioned in the original review do not reflect the implementation details and the empirical results. If the reviewer assesses that our rebuttal is unclear, or that there are open issues, we are happy to further address the open points in the remaining rebuttal time.
Response
Thank you for the detailed rebuttal and running those extra experiments for the Taylor approximation. My questions are all addressed. I have raised my score accordingly.
Decision
Accept (poster)