Strengths
1)
The notion of fully hyperbolic vision transformers is an important step in hyperbolic learning. Based on advances in learning with hyperbolic embeddings at the end of a network, fully hyperbolic learning can open up entirely new doors in the field.
2)
An open question in fully hyperbolic learning is scalability, as also pointed out in this work. Yet the paper manages to scale its experiments to the ImageNet-level despite operating in the Poincaré ball. Moreover, this is done with good scores. Scaling to ImageNet is a positive sign for fully hyperbolic learning.
Weaknesses
1+2)
The biggest issue with the paper is that most of the proposed components are not new. This limitation is coupled with extremely poor citation to existing literature. Many papers have in recent years investigated fully hyperbolic (visual) learning and have proposed neural network building blocks. Yet these are not cited.
The first contribution claimed in the introduction is that the authors proposed hyperbolic neural components. While they cite Hyperbolic Neural Networks by Ganea et al. (2018) in S2, they fail to mention this work throughout the method section. More importantly, Hyperbolic Neural Networks++ [a] is ignored. This paper not only outlines how to generalize MLPs and linear layers to the Poincaré ball, but even outline a self-attention algorithm in this space. Yet this manuscript does not mention the paper.
[a] Shimizu, Ryohei, Yusuke Mukuta, and Tatsuya Harada. "Hyperbolic neural networks++." ICLR (2021).
The related work on hyperbolic geometry in machine learning lists only 5 papers altogether and ignores recent work on fully hyperbolic convolutional learning, notably [b,c]:
[b] van Spengler, Max, Erwin Berkhout, and Pascal Mettes. "Poincare resnet." In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5419-5428. 2023.
[c] Bdeir, Ahmad, and Niels Landwehr. "Optimizing Curvature Learning for Robust Hyperbolic Deep Learning in Computer Vision." arXiv preprint arXiv:2405.13979 (2024).
In [b], a fully hyperbolic batch normalization is outlined, as well as linear layers, and more in the Poincaré ball. There are many more papers ignored, as summarized in [d,e]:
[d] Peng, Wei, Tuomas Varanka, Abdelrahman Mostafa, Henglin Shi, and Guoying Zhao. "Hyperbolic deep neural networks: A survey." IEEE Transactions on pattern analysis and machine intelligence 44, no. 12 (2021): 10023-10044.
[e] Mettes, Pascal, Mina Ghadimi Atigh, Martin Keller-Ressel, Jeffrey Gu, and Serena Yeung. "Hyperbolic deep learning in computer vision: A survey." International Journal of Computer Vision (2024): 1-25.
3)
The method outlines many small components but annoyingly never mentions which they claim as novel and which ones not. This is an issue since there is no citation in the method, except for one in the last sentence, making it impossible for the reader to understand which parts are copied from other papers and which ones are new. Below a summary of some issues:
- Hyperbolic linear layers have been proposed many times before, but not cited here.
- The ReLU and LayerNorm in Equation 10+11 are not hyperbolic, as the operations are performed in tangent space.
- The curvature in the positional embeddings (Eq 13) seems to just be a learnable parameter which has nothing to do with the curvature of the Poincaré ball.
- Self-attention in the Poincaré ball is given in [a].
- Residual connections in the Poincaré ball can be found in [b].
- Gradient clipping (Eq. 31) is common practice, see e.g. [f].
- Riemannian Adam (Eq. 32) is from Becigneul and Ganea (2019), although they are cited in Table 2.
As a result, the technical novelty of the paper is low, given that most components come from existing literature. This paper combines a number of them to create vision transformers.
[f] Dhingra, Bhuwan, Christopher J. Shallue, Mohammad Norouzi, Andrew M. Dai, and George E. Dahl. "Embedding text in hyperbolic spaces." arXiv preprint arXiv:1806.04313 (2018).
4)
The experiments are insufficient. Only 1 ImageNet result (Table 4) and 1 ablation study (Table 5) is given. Experiments on one dataset are however not enough. Moreover, multiple proposed components are never evaluated. In my view, the paper needs the following additional experiments to be more complete:
- Training and evaluation on other datasets. Can also be on smaller datasets.
- Starting from the pre-trained ImageNet model, fine-tune on smaller datasets and report test accuracy.
- Table 4 should also include runtime complexity and memory footprint. I expect that they will be much higher for HVT, but this is fine, it is good to be open about this.
- The geodesic regularization is not evaluated. What is its influence?
- Idem for layer scaling.