Trade-Offs of Diagonal Fisher Information Matrix Estimators

Summary

The authors study two popular estimators of the Fisher information matrix with respect to neural network parameters. They derive upper and lower bounds for the variance of these estimators and showcase them in applications to regression and classification problems.

Strengths

- Analyzing the convergence properties of Fisher information estimators is a timely and nontrivial endeavor, in part due to their use in natural gradient descent. - The paper is very extensive and mathematically sound.

Weaknesses

- The paper is highly technical and the numerical examples are fairly toy. - The paper takes strong inspiration from a previous work [Soen, Alexander, and Ke Sun. "On the variance of the Fisher information for deep learning." Advances in Neural Information Processing Systems 34 (2021): 5708-5719] on the topic. As such, I question the value the present submission adds to this line of research. In particular, I think the submission does a poor job in highlighting the central novel aspects of their work and reviewing the field.

Questions

- The authors should clarify clearly to what extent their submission improves upon previous works, particularly [Soen, Alexander, and Ke Sun. "On the variance of the Fisher information for deep learning." Advances in Neural Information Processing Systems 34 (2021): 5708-5719]. How do their bounds compare to the ones previously derived? - In my opinion, the impact of the submission would be substantially improved if the authors state more clearly in what scenarios which of the two estimators will be preferred and provide a rough guideline for practitioners. - Could the authors provide a numerical example for regression similar to Fig. 2?

Rating

5

Confidence

3

Soundness

4

Presentation

3

Contribution

2

Limitations

There is no further need for authors to address limitations.

Summary

The paper analyses two estimators of the Fisher Information matrix and specifically their variances, which (from reference 37) have closed-form but non-practical expressions. The authors show a sequence of inequalities and derive practical bounds for these variances, both element-wise and trace-wise, both conditioned to x and not. Several remarks and trade-offs on these bounds clarify when an estimator is preferable over the other. The results hold in the general exponential family likelihood but are also further concretized for regression and classification.

Strengths

The paper is overall very well written and structured. Notation is clear, results are properly presented and every step is further clarified in the appendix. \ The alternation between formal statements and explanations is great and makes the paper vey pleasant to read. I particularly like the amount of remarks and observations that discuss specific terms of the equations, explains them and connect them to each other. And the same is true for all the highlighted trade-offs (for example line 212-215)

Weaknesses

I have found no major weaknesses in the paper. I report some minor things. I personally dislike the notation $\hat{\mathcal{I}}_1(\theta_i)$ for the i-th element of the diagonal introduced in line 87. This notation suggest that this term only depends on $\theta_i$, but it actually depends on the whole $\theta$ vector. On the other hand, this is quite a light easy notations, and I can't think of alternatives which are not much heavier to read. Also, I find the definition in line 120 "is defined by $\hat{\mathcal{I}}_j(\theta_i)$ with $x=x_1=...=x_N$" to be yes understandable but not super clean. The alternatives are not as compact, but I'd recommend using the extra space to clarify this a bit better. At least you should write that $y_1,...,y_N$ are i.i.d. from $p(y|x;\theta)$ and not from $p(y,x;\theta)$ as in $\hat{\mathcal{I}}_j(\theta_i)$. It would be nice to explicitly write the proof for the variance closed expression in Lemma B.1, yet clarifying that it's not a paper contribution. This is for two reasons: (1) it would make the paper self contained, without the need for the reader to read reference 37 and (2) the notation is slightly different and it would be easier for the reader to be consistent.

Questions

In line 168. Shouldn't "of Eqs. (4) and (6)" instead be "of Eqs. (4) and (5)"? And consistently shouldn't "$\mathcal{V}_1(\theta_i|x)$" be "$\mathcal{V}_2(\theta_i|x)$"? \ I ask this because "small shifts in parameter space yield large changes in the output" to me refers to the magnitude of the network jacobian $||\partial_i h(x)||$, and that appear in the bound for $V_1$, not for $V_2$. Am I misunderstanding something or is this a typo? In line 246. Isn't the bound sample complexity $\mathcal{O}(\frac{1}{N_x} + \frac{1}{N_y})$? I can't see why it should be the product, can you elaborate more on this derivation?

Rating

8

Confidence

5

Soundness

4

Presentation

3

Contribution

3

Limitations

I think it would be nice to clarify better that the work is incremental on "Soen et all - On the variance of the Fisher information for deep learning" [reference 37] and their closed form expression reported in the appendix in Lemma B.1. (also it would be nice to move these closed form expressions to the main paper).

Summary

In this paper, the authors analyzed two different estimators, $I_1(\theta)$ and $I_2(\theta)$, for the diagonal elements of the Fisher information matrix of a parametric neural net model. These diagonal elements are approximations for the entire matrix, which is unfeasible to be calculated in real neural nets models. They identified the situations in which each of these estimators is preferable with respect to the other.

Strengths

The Fisher information matrix is a major parameter driving the quality of the fit and highlighting important aspects of the interdependence between the parameters. It has been used in several papers of continual learning, the research area that I follow. It focuses on the diagonal elements of the Fisher information matrix, offering approximations that are feasible to calculate in real neural net models, addressing a significant practical challenge given the unfeasibility of computing the entire matrix. he authors identified specific situations where each of the two estimators, $I_1(\theta)$ and $I_2(\theta)$, is preferable. This practical guidance can help researchers and practitioners make informed decisions about which estimator to use in different scenarios.

Weaknesses

This is a highly theoretical paper and I have difficulty following all the findings presented by the authors. One of the main problems with this paper is that it has a large intersection with Soen and Sun (2021) and this becomes clear while we read the paper. In the Related work, Soen and Sun (2021) is mentioned in passing and it is a major reference work for the current manuscript. It is worth to have a better explanation of the relative merits of these two papers. Both papers analyzed two different estimators, $I_1(\theta)$ and $I_2(\theta)$, for the Fisher information matrix of a parametric neural net model. While the older paper focuses on the matrix as a whole, the present manuscript focuses on the diagonal elements. However, the work of Soen and Sun (2021) is repeatedly referred to during the technical development of the paper. The contribution of this paper seems marginal when compared to the previous paper but still relevant. The results are derived assuming $\theta$ is equal to its true value. In practice, the maximum likelihood estimator (MLE) of $\theta is used, which introduces additional variability. The paper does not adequately address the implications of this assumption or the consequences when using the MLE, which limits the practical applicability of the results. The paper does not discuss the impact of model misspecification on the results. In real-world scenarios, the data might not follow the proposed model exactly, and the inverse-Fisher information matrix might not represent the correct asymptotic variance of MLE estimators. The authors should comment on how their findings would be affected in such cases.

Questions

The results are all given using $\theta$ equal to its true value. In practice, we use the MLE estimator of $\theta$. The results are not valid anymore as we have this additional source of variability. The authors may comment on the consequences of this?

Rating

5

Confidence

1

Soundness

2

Presentation

2

Contribution

2

Limitations

This is a theoretical paper with no immediate connection to societal impact.

Summary

Summary ------- The paper studies two estimators for estimating the diagonal of a Fisher information matrix (FIM) of a parametric machine learning model. Both estimators are based on equivalent expressions for the FIM. The first is the standard definition, expressed in terms of the derivatives of the log likelihood, and the latter, expressed in terms of the second derivative when the model is twice differentiable with respect to the parameters. The estimators replace the expectation with an average over the data. The authors provide bounds on the variance of both (unbiased) estimators in terms of the minimum/maximum eigenvalues of the FIM, and then provide bounds on these eigenvalues under special cases. They also provide a modest empirical evaluation. Overall, while I think there is a fair amount of technical meat in the paper, I am unconvinced about the problem. I will wait to hear from more expert reviewers in this field before I make my decision. Detailed comments: ------------------ While I am theoretically inclined, I am not an expert in this sub-field. Therefore, I will keep my comments at a high level. Significance of problem: I struggled to understand when estimating the FIM will be useful. - One point raised by authors is that it is used in optimizing NNs. If this was true, it would have been nice to see the estimators used in NN optimization empirically, or a theoretical analysis showing how the bounds derived here percolate to bounds on optimization. - Another explanation given is that it could be used to analyze the structure fo NNs. But this is stated vaguely, and it wasn't clear to me that such an analysis would help design better neural networks. Significance of techniques: The proof in the appendix is substantial but the authors have not stated which techniques used here are novel. Currently technical results are presented one after the other. It might help the reader/reviewer appreciate the contributions of the paper if the authors can highlight the novel tools they used to derive these bounds. Presentation and writing: The technical content was clear for the most part. However, the writing was very equation-driven. I understand that this is in part due to the nature of the paper, but the authors should simplify the exposition further. One option is to cut down on some of the technical content.

Strengths

See above

Weaknesses

See above

Questions

See above

Rating

6

Confidence

2

Soundness

3

Presentation

2

Contribution

2

Limitations

See above

Keep score

Thanks for the clarification. While I do agree with the other reviewers that the improvements made on top of [37] could have been clarified better, I do think that they are overall clear and valuable. Hence, I keep my score and strongly support acceptance.

Response

Thanks for responding to my questions. Based on the reply, I have increased my score.