Summary
The paper carefully examines the effectiveness of various Hyperbolic Neural Networks (HNNs) by measuring their Hierarchical Representation Capabilities (HRC), an evaluation process named as Hierarchical Representation Capability Benchmark (HRCB). The four metrics developed for HRCB include Root Node Hierarchy, Coordinate Origin Hierarchy, Parent Node Hierarchy, and Sibling Node Hierarchy, which altogether measure how well the hierarchical structure is embedded in hyperbolic space. The paper also proposes pre-training strategies upon improving a model's HRC, and empirically assess the relationship between HRC and downstream performance. Experimental results show that HNNs' HRC has a significant impact on downstream performance, and pre-training HNNs towards enhancing HRC can improve its performance.
Strengths
- [S1] The motivation of looking back upon hyperbolic neural networks and closely assessing their effectiveness on downstream tasks with respect to theoretical work is very interesting, and I believe any insights would greatly benefit the geometric deep learning community.
- [S2] The scope of experimentation is fairly comprehensive, covering various downstream tasks, hyperbolic manifold spaces, and neural network architectures.
Weaknesses
- [W1] **The overall contribution is not clear within the writing and experiments.** Is the paper hoping to show why HNNs are effective? or that HNNs do not achieve theoretically optimal embeddings?. From the experiments, it seems the goal of the paper is that 1) there exist a gap in HRC between existing HNNs and theoretically optimal embeddings and 2) reducing the gap via pre-training helps boost downstream performance, yet the overall writing (e.g., abstract and introduction) makes it confusing on what to expect from the experiments: reasons behind HNNs' effectiveness or limitations of HNNs.
- [W2] **The presented empirical observations in the text are unclear and somewhat misleading.** For instance, page 7 discusses how the LR target for NC "does not need to distinguish the position relation" among nodes, yet overfitting on LR helps improve HRC, which seems counterintuitive. Why is this the case? Furthermore, page 8 mentions how "within the applicable scope of HNNs, performance can be improved by enhancing HRC", with the node classification task being "out of scope". This is misleading considering that many previous work have shown performance boosts in node classification by leveraging hyperbolic models [A, B, C].
- [W3] **The figures showing Friedman test and Nemenyi post-hoc tests are extremely hard to read.** It would be better to categorize results based on what the authors are hoping to convey through the experiment: as an example, for Figure 6(a), if the main observation is that GD, HR, and FD help HNNs learn position relation unlike LR, it would be better to simply draw a bar chart (or multiple bar charts, one for each manifold) with targets on the x-axis and the HRC values on the y-axis. That way, we can visually observe the orderings currently written as text within the plots.
- [W4] **Downstream performance results are missing exact scores and are only compared in terms of rankings.** For Subsection 5.2.3, it would be better to simply present the downstream results in exact scores using the scoring metrics for each downstream task (F1 score for node classification, mAP for graph reconstruction and so on). This way, we can concretely estimate how much better/worse each method performed compared to another, and whether the results are within reasonable range compared to existing literature.
[A] Chami et al., Hyperbolic Graph Convolutional Neural Networks. (NeurIPS 2019)\
[B] Liu et al., Hyperbolic Graph Neural Networks. (NeurIPS 2019)\
[C] Zhu et al., Graph Geometry Interaction Learning. (NeurIPS 2020)
Questions
- [Q1] The Coordinate Origin Hierarchy metric $M_o$ seems to assume that the root node is located near the origin, while this constraint is not made explicitly during training of HNNs. Considering that Combinatorial Construction [D], on the other hand, trivially satisfies this assumption, would you still consider using this metric to be valid for fair comparison?
- [Q2] Is the L strategy described in the beginning of subsection 5.2.3 equivalent to adding a weighted auxiliary loss to the downstream predictive loss? This seems very similar to how the HGCN paper used a link prediction regularization objective for node classification [A]. Thus adding a few discussion on this connection and giving the strategy a proper name rather than just "L" could help towards better clarity.
[D] Sala et al., Representation Tradeoffs for Hyperbolic Embeddings. (ICML 2018)
Rating
3: reject, not good enough
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.