Summary
This work empirically studies the connection between the generalization gap and the fractal dimension of a neural network's optimization trajectory. A recent line of research has established such connections in a theoretical manner through upper generalization bounds. Specifically, three types of experiments are performed
1. An analysis of the correlation between fractal dimensions and empirical generalization gap in a typical training scenario. Here, the experiments from prior work [1] are replicated and analysed to greater detail.
2. An analysis of such correlations for models that do not generalize properly, obtained from training starting at an adversarial initialization.
3. A double descent behavior of fractal dimensions is observed when training models of varying width.
In the first two experiments, only few statistical significance of the correlation between fractal dimensions and generalization is observed, i.e., less than between generalization and the L2 norm of parameters.
Strengths
- The empirical evidence presented in prior work is closely examined and questioned. While this might not be as appreciated as the derivation of novel theoretical results or of previously unobserved empirical findings, it is definitely necessary in order to establish a solid foundation future work can build upon.
- There is substantial effort in identifying and transparently communicating the limitations of this work, much more than in all other submissions I have reviewed for NeurIPS 2024.
- Multiple different correlation scores are computed, and the findings are tested for statistical significance.
- The background section provides a good overview over the research on generalization bounds in terms of fractal dimensions.
Weaknesses
- In section 4.1, the experiments from [1] are carefully replicated. However, the observations do not match the ones from [1]. For example, for FCN-5 on CHD, a *negative* correlation score of $\rho = -0.7622$ is reported, whereas [1] reports a *positive* correlation of $\rho = 0.87$, which looks plausible from [1, Figure 2]. This inconsistency is surprising, but it is neither discussed nor mentioned.
If the setting is indeed equal, then the correlation coefficients should be much more similar. I do trust both works in that they correctly report their results, but this contradiction needs to be resolved in some form.
- This submission presents the empirical data, but is very cautious in interpreting it. On the one hand, it is good to let the data speak for itself, but on the other hand, the relevant part is how the observations relate to the larger context.
For example, at the end of section 4.1 (Correlation Analysis), it is stated that "Using this method, we conclude that the correlation coefficients between generalization and the norm or the ratio of learning rate over batch size are indeed significantly higher than the correlation coefficients with PH dimensions [l.209]"
What are the conclusions from this? Similar questions apply to the other sections.
- It seems that the data is collected from only 10 random seeds (in each setting). For a purely empirical work that seems a bit low, especially if strong claims are deduced from it, e.g. "We demonstrate that fractal dimension fails to predict generalization of models trained from poor initializations"
- Not a weakness of the submission (rather on my end), but worth mentioning: I do not have sufficient knowledge on testing for correlations or independence. Thus, I will not comment on the used statistical pipeline (setup and choice of correlation coefficients, null hypotheses, tests, etc)
- typos:
l.290 we were only work ...
l.291 preventted
### References
1. Dupuis et al., Generalization Bounds using Data-Dependent Fractal Dimensions, ICML 2023.
2. Birdal et al., Intrinsic Dimension, Persistent Homology and Generalization in Neural Networks, NeurIPS 2021
3. Hofer et al., Connectivity-Optimized Representation Learning via Persistent Homology, ICML 2019
4. Chen et al., A Topological Regularizer for Classifiers via Persistent Homology, AISTATS 2019
5. Carrière et al., PersLay: a neural network layer for persistence diagrams and new graph
topological signature, AISTATS 2020
Questions
- Your empirical observations contradict the theoretical predictions deduced from the generalization bounds in [1,2,..]. What do you think is the cause? Is it, because there is an error in the theory, are the assumptions in the theory unrealistic (not fulfilled in practice), is the fractal dimension not the decisive term in the generalization bounds, etc?
- A limitation that is not stated in your discussion section, but most likely plays a role, is the uncertainty of the fractal dimensionality estimates. I wonder, how much difference there is, if you continue SGD for another 5k iterations and re-estimate the dimension from them. Are the estimates approximately equal? Is their order the same? Is there any rank correlation between estimates from the first 5k and the second 5k iterations at all? Note that even if the estimates are equal, it does not mean that the estimates are accurate, as they might be biased.
- I guess, this was not your intention, but the remark "While efficient PH computation is a field in itself, integrating it with network training to make fractal dimension computations more feasible is an interesting direction, especially if we are able to derive a regularization term. [l312]" suggests that integrating persistent homology into neural network training hasn't been done before. This is not the case, and there are *many* works that do so, [3-5] to name a few. In fact, such a regularization term is even used by Birdal et al. [2] to encourage low ph dimensions.
- I did not understand your test of conditional independence, cf l.257. Would you explain it to me?
Limitations
Yes, the authors adequately addressed the limitations!