SmoothHess: ReLU Network Feature Interactions via Stein's Lemma

Summary

For smooth functions the joint interaction between pairs of features and their impact on the function output can be modeled and quantified via second order partial derivatives. The Hessian of a neural network can therefore provide insight as to how interactions between pairs of features impact predictions made by the network. ReLU networks are piecewise linear and therefore have a zero Hessian almost everywhere, as a result a different approach to quantifying interactions is needed. One way to overcome this issue is to consider a smoothed approximation of the network. Prior works substitute ReLU for Softplus and compute the Hessian of the corresponding smooth network, however, the sensitivity to the $\beta$ parameter of the Softplus activation is pronounced. Furthermore this method affords poor control over the level of smoothing. This work considers instead estimating interactions by computing the Hessian of the convolution of the network with a Gaussian - this Hessian is referred to as SmoothHess. Furthermore, using Stein's Lemma it is shown that one can efficiently estimate SmoothHess from first order information alone using Monte Carlo simulation. Non-asymptotic bounds on the error of this estimate are also provided in Theorem 3. Experimental results demonstrating improvements of this method over existing techniques, at least in the context of relatively simple problems such as MNIST and CIFAR10, are provided.

Strengths

This paper appears to provide a new and natural method with which to estimate the impact of interactions between pairs of features on the network output in the setting in which the network of the Hessian is zero almost everywhere. The method seems principled and outperforms existing methods on a number of benchmarks. The technical results appear correct although I confess I did not check them thoroughly. The paper is generally well written and clear. I am not an expert in this space so cannot say much to as to its broader context with respect to prior works.

Weaknesses

One minor concern might be around how one actually chooses the covariance matrix in practice. However, choosing hyperparameters is a problem for the other methods as well it seems and furthermore this method appears to give you more direct control of the smoothing.

Questions

1. Are these techniques limited to ReLU networks or could one in fact apply them to a more general class of functions for which the first and second order partial derivatives are not defined everywhere (non-smooth) and or zero almost everywhere? 2. How does one choose the covariance matrix and hence the degree of smoothing? 3. Can you identify any (potentially pathological) examples in which SmoothHess really fails to model interaction effects? In short could you provide further comment on potential failure modes of the method and highlight any warnings one should be aware of when using it?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Limitations of the work, in particular the computational complexity with regard to the input dimension, are highlighted and discussed in the conclusion. A link to code is included in the supplementary material. Potential social impacts are also discussed in the supplementary material: the authors are careful to point out that as with any method used to explain decisions made by a model one needs to treat such outputs with care and due oversight, and that it is best to work in tandem with domain experts.

Summary

The goal of this paper is to model the point-wise interactions between input features in a ReLU network. ReLU networks are piece-wise linear, which inhibits using Hessian to model feature interaction (since it’s zero almost everywhere). This paper proposes SmoothHess, a new method to compute feature interaction by the Hessian of a smooth surrogate of the ReLU network. The authors prove that SmoothHess can be estimated using only gradient calls, and prove non-asymptotic sample complexity bounds. One result that’s particularly elegant is the connection between SmoothHess and SmoothGrad: combined, they define a local, second-order Taylor expansion of the network. The author designed several evaluations of the proposed method based on this connection. The main baseline that SmoothHess is compared against is SoftPlus Hessian which is based on an alternative smooth surrogate of ReLU. These methods are evaluated as local second-order estimates of the ReLU model: they are evaluated by the accuracy of their local approximation, as well as the capability to guide adversarial attacks. SmoothHess performs best in both tasks on three standard image classification datasets. The authors also performed qualitative analysis on a regression problem from the medical domain.

Strengths

Although feature interaction is a well-studied topic, I find the proposed method to be quite novel. The motivation of the problem, i.e., the piece-wise linearity of ReLU networks is very clear and the survey of related work seems thorough. The proof sketches provided in the main paper are explained clearly, with detailed references to pieces of proofs from other work. Overall, I find the paper to be very well-written and easy to follow. It strikes a great balance between having detailed theoretical results and clearly conveying the intuition.

Weaknesses

The main weakness of the paper is the comprehensiveness of experiments. Due to computational cost of computing SmoothHess, all experiments are conducted on relatively small-scale datasets and small-ish models. Although I do not see reasons why this method would not work on larger models, it would make the paper much stronger to have empirical evidence. I also find the qualitative analysis a bit dry. Part of this is due to my lack of familiarity with the Spirometry regression task. But I cannot confidently evaluate the significance of that result, in particular regarding what it means for interpretability of ReLU networks in general. Can this method improve tasks that can benefit from interpretability, like error detection, trust calibration, debugging, auditing, or knowledge discovery? I'm not sure. This is a bit disappointing since interpretability is a primary motivation for this work. It would help to at least have some qualitative comparison (and with more competing methods) on more familiar datasets to at least help us understand the behavioral differences between SmoothHess and existing methods.

Questions

Regarding Equation 10: if my understanding is correct, what Equation 10 does is it defines how SmoothHess (and competing methods) can be used in conjunction with gradient methods to define a local, second-order model of the underlying ReLU model. This equation is then used as the basis of experiments which evaluates the goodness of fit from several aspects. Is this understanding correct? Are there alternatives to Equation 10? Is there an alternative way to do similar experiments without Equation 10? Is it possible for this specific definition (for example by using f instead of g as the first term) to bias the results?

Rating

8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

4 excellent

Limitations

The authors have adequately address the limitations.

Summary

This paper introduces SmoothHess, a method to define a smooth approximate Hessian for networks where many components of the Hessian may be zero (e.g. ReLU networks). This can aid in tasks where interactions between features are needed, but where those components of the exact Hessian are zero. SmoothHess is defined as the convolution of the output of the network with the Gaussian. It is easy to show, using Stein's lemma, that this is equivalent to the expectation of the gradient of a perturbed output over that Gaussian distribution. They then do a number of experiments. In the first two sets of experiments (perturbation MSE and adversatrial attakcs) on three datasets to show the advantages of using SmoothHess over alternative smoothings of ReLU, such as softPlus. In both experiments, SmoothHess significantly outperforms SofPlus. They also test this on a 4 quadrants dataset with similar results. They also apply this to a real-world medical dataset of spirometry to measure feature interactions.

Strengths

The paper shows the benefit of using a smoothened Hessian of ReLU networks for tasks involving feature interactions. The benefit of the method is that it is post-hoc and can be applied to the output of a trained network without a modification to the architecture. The theoretical derivations are easy to follow and well organized. The experiments also make a good case for using their construction, showing superiority to SoftPlus. Overall, the simplicity of the method is its main strength.

Weaknesses

My main reservation is how impactful the work can be. It is a neat idea, but with a rather limited scope. For example, given the expense of computing SmoothHess and the fact that we only need it for diagonal blocks of the Hessian of ReLU networks (explained next) makes the impact rather limited for me. Maybe the authors can explain or emphasize more the breadth of the problems where this would be useful or required? One minor issue to raise is that, as I understand it, the Hessian of ReLU networks is *not* zero almost everywhere. In multi-layer networks, only the Hessian components involving weights in the same layer are zero. All cross layer Hessian terms should be nonzero. Let $W^l_i$ denote a component indexed by $i$ in the weights of layer $l$. The Hessian term $${\partial^2 F(x) \over \partial W^l_i \partial W^k_j} $$ is in general not zero, even in ReLU networks. In other words, the paper's claims about vanishing Hessian only hold for block diagonal components of the Hessian. They need to argue for why these components are more important than off-diagonal blocks. Additionally, the baselines in the experiment are rather limited. SoftPlus is not the only smooth alternative to ReLU. The swish activation and similar functions could have been compared as well.

Questions

1. Why only consider diagonal Hessian components, while off-diagonal blocks of ReLU networks can be non-zero? 2. How do baselines other than SoftPlus (e.g. swish) perform? 3. In the spirometry experiment, what is the conclusion? Can you trust or validate that the interactions revealed by SmoothHess are real? Does it help predict which datapoints to discard, or is it about which ones not to drop? 4. regarding the quadratic computation of SmoothHess, is there an alternative for feature interactions which is not quadratic?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

The paper does discuss limitations, notably the quadratic cost in feature dimensions, which is prohibitively expensive for large models/datasets. I don't see an immediate societal impact issue.

Summary

The paper propose to compute Hessian of a ReLU neural network by a convolution of the network function with a Gaussian distribution function. It's equivalent (or cover) some prior work, e.g., SmoothGrad. The method is generally easy to understand, without modifying the network structure. A non-asymptotic bound on the sample complexity is provided.

Strengths

Strengths: -- The proposed smooth-surrogate is easy and straightforward to understand, the computation method is somewhat novel using Stein's Lemma, following [40] with a mild extension to all Lipschitz continuous functions, which addresses the challenge posed by the zero Hessian of piecewise-linear ReLU networks. The paper provides a non-asymptotic bound on the sample complexity of the estimation procedure. -- Proposed SmoothHess can be applied post-hoc and does not require any modifications to the ReLU network architecture. This allows analysis and interpretation of the interactions of existing ReLU neural networks without retraining or redesigning the model. -- Efficient sampling: SmoothHess utilizes an efficient sampling algorithm, which only requires network gradient calls, making it computationally feasible to large-scale networks and complex datasets.

Weaknesses

Weaknesses: -- Scope limited to ReLU networks: While SmoothHess addresses the challenge of zero Hessian in ReLU networks, its applicability is limited to this specific activation function. It would be useful to discuss the potential generalizability of the method to other types of neural networks with different activation functions. -- Generalize beyond simple datasets: SmoothHess can be applied to large models and datasets with its simple computations, it would be beneficial to clarify the generalizability of SmoothHess and the potential limitations in terms of dataset diversity.

Questions

Some notations are confusing, e.g., caption of figure 2, is it $\log(10\sigma)$ or $\log_{10}{\sigma}$? also the text, "Aside from at minute", what does "minute" mean here?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

2 fair

Contribution

2 fair

Limitations

See above

Summary

The work proposes an approach for measuring feature interactions for neural networks that have no useful second order gradients or Hessians (notably those employing ReLU). The approach convolves a smooth normal with the network as a proxy with usable Hessians and then offers a means of estimating them using sampling of first-order gradients, thereby making it more tractable than methods that rely on computing Hessians naively. Experiments evaluate the proposed method and several adaptations of existing methods for computing 1) the 3rd-term (second order gradient) of the Taylor expansion of the output of a neural network around an input, and 2) for adversarial inputs via optimization using (1). In these experimental terms, the proposed approach (aided by SmoothGrad for the 2nd Taylor term) performs better than those based on only the first order gradient or those that use include both first and second order gradient but proxy ReLU models using SmoothPlus activation.

Strengths

+ Almost strictly better experimental results than the compared-to methods. + Hessians for networks which don't naively have them due to technicalities should be generally useful in a wide variety of applications.

Weaknesses

- Experimentation focuses on agreement between a network and one evaluated as per Taylor expansion using SmoothHess. While this may be a fairly general test, it does not offer any insight into benefits in cases where the method would actually be useful to use, with no trivial alternatives. Presumably in most cases we can already evaluate the network regardless of Hessian non-existence so at surface level there is no need for Taylor expansion and thus the Hessian of the smooth proxy. The adversarial attack is also based on use of Taylor expansion of smoothed model as proxy so it does not improve this problem. The final case-study does present interaction results but is difficult to follow (see questions below). I suggest, then, that the paper include evaluation of the methods that focus on the Hessian, instead of network output. One example is to test adversarial attacks on gradient-based attributions by gradient descent on the gradient (i.e. using Hessian). Compare to other methods for attacking attributions. - Tradeoffs with respect to the parameters are not explored or described. The use of smoothing serves to increase the impact of network regions further away from each point but smoothing is a double-edged sword. If smoothing (and subsequent sampling) is insufficient compared to the sizes of linear network regions, the results may not be useful. On the other hand, large enough smoothing reduces the network to a constant. One impact of this tradeoff is how fast gradient descent can arrive at good solutions. Thus in addition to the above suggestion of testing usefulness of the Hessian, I also suggest that the effect of the parameters (or smoothness) have on that usefulness. For example, sufficiently significant smoothness I expect to result in useless gradients of gradients for attacking attributions. Smaller things: - What is δ (without subscript) in Equation 8a? Did you mean δᵢ ? - On line 117, the expectation and perturbation are presumed not significant to the point that the Hessian is 0 almost everywhere. - By "second-order interactions" did you mean interactions via second-order gradients? Interactions seem already second-order so saying second-order interactions is suggesting third-order gradients. - In Table 1, are the two best (bolded) results in the 4th numerical column identical or is it just that the printout identical? - Around line 255, you write "We denote Δ* ... as the output after x₀ is attacked". This may be a bit confusing as I think you mean Δ* as the perturbed input, not the network output. Is that right? - Line 349 writes "Interestingly, the smaller kernel width constraints the interactions to features within a small locality." Why is that interesting? Isn't that expected? - Typo near "between between". - The Adversarial Attacks paragraph on line 254 and Equation 12a is missing the target class of the attack.

Questions

- Conceptually, when sampling as per MC-estimation, if samples end up not spanning more than 1 linear region of the original network, would the estimate be zero in the same sense as second order gradient is zero? - What is the impact of smoothness/covariance on sample bound as per Equation 9? - Does "SP H + G" mean "SoftPlus (Hessian + Gradient)" or "(SoftPlus Hessian) + (ReLU Gradient)"? Line 285 suggests the latter but the label/caption suggest the first. If the first, the comparison may not be as fair as it could be. - Example of Figure 3 is difficult to understand. Why are the arrows labeled "interaction" crossing from one method to another? Also, SmoothGrad vs. SmoothHess is indicated by both color and the separate graphs? Did you mean that green/red are indicators of positive or negative interactions for both of those methods or for one respectively? Or perhaps the second of each pair of graphs shows the interaction of the other timesteps with the first timestemp only? Also, what is the kernel width for the example shown in Figure 3? - The covariance matrix is set in a specific way in the text but the benefit of the option to set it otherwise are mentioned. How would one go about setting it alternatively and for what purposes?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

The work describes experiments with adversarial attacks which could use some mention with relation to negative societal impact.

Summary

The author introduces a novel method to approximate the Hessian of a smoothed loss function at a specific point. By incorporating a variation of Stein’s lemma, they cleverly estimate the gradient over a smoothed surrogate of a ReLU network at that point. To gauge the method’s effectiveness, they measure its performance by testing the normalized L2 distance between the loss function and its first- order and second-order approximations within a ball centered around the given point.

Strengths

The method offers a straightforward and comprehensible approach, presenting a powerful and novel tool for approximating the Hessian of a ReLU network at a specific point. The author’s analysis is comprehensive and remarkably easy to follow. The experiments conducted in the paper appear relevant, and the obtained results show promising potential.

Weaknesses

While the ideas and methods of the paper are solid, I came across some read- ability issues. 1. The statement of Theorem 3 (line 211) is confusing. The real positive $\delta$ is used without declaration, while the random variables $\delta_1,\ldots,\delta_n$ are declared. This makes the reader wonder how one can compare a probability with a random vector, and what is the meaning of this inequality which is essentially a random variable itself. I had to read the proof in the appendix to fully understand the statement of the theorem. I think using a different symbol for $\delta$ might increase readability. 2. I would add the definitions of the oracles used in the article for clarity.

Questions

1. line 103 "An general L-hidden..." is a typo? 2. line 287 "with the non-smooothed..." is a typo?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

3 good

Contribution

3 good

Limitations

The article address the time and space complexity of the method in the presence of a high-dimensional input.

Re: Supplementary Clarifications

In the Official Rebuttal, we have referenced a comment we would be making below our official response to the Reviewer, clarifying more points, which we have already prepared. However, upon further re-reading of the guidelines and new 6,000 character limit rule, we are now uncertain if providing this would be a violation of any policy and thus decided not to post. Thank you for your thoughtful questions, which we have tried to answer within the given character limit. If you have any specific questions please feel free to reach out during the discussion phase and we would be happy to respond.

Thanks fro clarifications

Thank you for clarifying which Hessian you are using, which I had misinterpreted. For Hessian w.r.t. the input, it is clear that off-diagonal terms are zero for ReLU networks. Also, thanks for the Swish experiments. I will increase my score.

Re: Additional Information

**We kindly thank the Reviewer for their response. We hope that this comment will address any remaining concerns; if not, we are happy to provide any additional information as needed.** --- ## Smaller things **S1, S7 Typo** We will change to $\delta_i$, and remove the second between, in the final version. **S2 Expectation** The space where the Hessian is 0 has measure 1 under the Gaussian distribution, from here the expectation in line 117 follows. **S3 Second order** We mean second order interactions via second order gradients. We are using the terminology: X-order interaction conveys an interaction between X features [1]. **S5 Adv. attack phrasing** Thank you for catching this. We will change to “the perturbed input” in the camera ready version. **S6 Kernel width** We agree that this behavior is to be expected. We mean that this provides some qualitative evidence that the interactions SmoothHess picks up “make sense”; i.e., it is following the intuition you have for why this result is to be expected: smaller kernels result in a smaller receptive field for neurons in each layer. We will clarify this in the camera-ready version. **S8 Missing target** $\tilde{f}$ in Eq. 12a is a real valued function: the Taylor expansion around the predicted class SoftMax score for the point $x_0$ (lines 261-262). Thus, no target is required in Eq 12a. We will make this more clear in the final version. --- ## Questions **Q1 One region** Let us assume we are estimating SmoothHess for a point $x_0 \in \mathbb{R}^d$ and function $f : \mathbb{R}^d \rightarrow \mathbb{R}$ using covariance $\Sigma \in \mathbb{R}^{d \times d}$ . We have $N$ IID samples $\delta_1, \ldots, \delta_n \sim \mathcal{N}(0, \Sigma)$. Further, each perturbed point $x_0 + \delta_i$ lies in the same linear region $Q \subseteq \mathbb{R}^d$. Our (unsymmetrized) estimator would be $\hat{H}^\circ_n(x_0,f, \Sigma) = \frac{1}{n} \sum_{i=1}^n \Sigma^{-1} \delta_i [\nabla_x f(x_0 + \delta_i)]^T$. Now, because we have assumed each $x_0 + \delta_i$ lies in the same linear region, the gradient of $f$ is the same at each point: $\nabla_x f(x_0 + \delta_i) = G, \forall i \in \{1, \ldots, n\}$ for some constant $G \in \mathbb{R}^d$. Thus our estimator may be written as $\hat{H}^\circ_n(x_0,f, \Sigma) = (\Sigma^{-1}) (\frac{1}{n} \sum_{i=1}^n \delta_i)(G^T)$. Here the terms in the left and right parenthesis are constants. Note the term in the middle parentheses is the sample mean of an R.V. distributed from $\mathcal{N}(0,\Sigma)$. Thus, if one has many samples the middle term will be close to, but not necessarily exactly, $0$ and accordingly so will $\hat{H}^\circ_n(x_0,f, \Sigma)$. It is not necessarily the case that $\lim_{n \rightarrow \infty} \hat{H}^\circ_n(x_0,f, \Sigma) = 0$, as eventually one would sample points in different regions: $x_0 + \delta$ s.t. $\nabla f(x_0 + \delta) \neq G$. However, It can be seen from this formulation that SmoothHess would give the $0$ matrix, in the infinite limit of samples, for a linear network $f$. **Q2 Bound** We leave the explicit incorporation of the covariance into our bound to future work. **Q4 Figure 3.** SmoothHess can be characterized as a second-order extension of SmoothGrad, and SmoothGrad values can be calculated at the same time as SmoothHess with minimal additional computational cost. Therefore, in the Spirometry experiment we use SmoothGrad to first identify first-order feature importance values, and then use SmoothHess to identify second-order interactions. In Figure 3, the left/right barplots for each spirometry sample represent the SmoothGrad/SmoothHess (respectively) attributions for 0.5sec blocks of features. In our example we are interested in the interactions for the first 0.5sec feature block, which is identified as being a top feature using SmoothGrad. Therefore the SmoothHess plot shows interactions between the first 0.5sec feature block and every other feature (i.e. the first row in the $d \times d$ feature interaction matrix). The red arrow points to the top interacting feature (arrow endpoint) with respect to the first feature (arrow source). The red/green bar colors are meant to indicate positive/negative values of SmoothGrad and SmoothHess; we will change this to a single color to reduce confusion. The kernel width for the example in Figure 3 is 200; this and other model details are listed in Appendix E.1. Thanks for your feedback, we will add detail to the figure caption to make this more clear. **Q5 Alt. cov.** We kindly direct you to paragraphs 4-7 of response W1 to Reviewer bZYQ for the answer to this question. ## References [1] Tsang, Michael, Dehua Cheng, and Yan Liu. "Detecting statistical interactions from neural network weights." arXiv preprint arXiv:1705.04977 (2017).