Summary
The paper generalizes the ResNet layer to non-euclidean geometries by replacing the Euclidean sum with the exponential map. The theory is general and applies to any smooth metric. They propose a way to parametrize a vector field on the manifold which is more geometrically principled than the trivial vector field embedding. Empirically they show improvement in performance on hyperbolic datasets and PSD spaces compared to some baselines.
Strengths
The paper theory is general and applies to any smooth manifold metric. In ResNet a neural network produces a vector field and the output of the ResNet is the input plus such vector field. The proposed generalization, assuming access to a vector field on the manifold, defines the output of the ResNet as the exponential map of the input in the direction of such a vector field. This is consistent with the Euclidean case.
A vector field in Euclidean space can simply be the output of a neural network, while on a manifold more care is required. Using an embedded vector field is straightforward, but, as correctly pointed out by the authors, it is not very principled geometrically. The authors propose a computationally tractable and geometrically principled way of defining a parametric vector field on a manifold. The idea is to make use of a collection of $\mathbb{R}$ valued functions (obtained as a projection on hyperplanes or similar) to define, through push forward and pullback, a vector field on the manifold.
Extensive specific examples of such a collection of functions are given for the hyperbolic and PSD manifolds. Experiments are also performed in these cases, and the proposed approach appears to outperform the current state of the art.
In the general manifold case (Appendix B.5), the idea of projection on pseudo-hyperplane is appealing and well-argued. And the further generalization to “hyper-disks” allows proper formal extension also to the non-geodetically-complete manifolds.
Weaknesses
The definition of the vector field (Appendix B.4) is not sufficiently formal and contains mistakes. Specifically, the author assumes access to a smooth function $f:M\rightarrow \mathbb{R}^k$, a so-called “feature map”. The differential $D_x f$ is then a linear map from the tangent space in $x$, $T_x M$ to the tangent space in $f(x)$, $T_{f(x)} R^k = R^k$. Observing that the dual of $R^k$ is isomorphic to $R^k$ itself, the pullback of the differential $(D_x f)^*$ can be seen as a map from $R^k$ to $(T_x M)^*$, for every $x\in M$. This function is evaluated in $f(x)\in R^k$ such that the vector field (as defined in line 215) is a map
$$ l_f: x \rightarrow (D_x f)^*(f(x)) $$ which, as we saw, take values in $(T_x M)^*$ and NOT on $T_x M$, as line 215 is saying.
We would like to see this inconsistency explained. Is this based on the observation that both $(T_x M)^*$ and $T_x M$ are isomorphic to $\mathbb{R}^{dim(M)}$?
And also, can you reason about the choice of evaluating the pullback of the differential (line 215) in $f(x)$? Is this somehow principled?
The equivalence with Euclidean ResNet shown in Appendix D is a proper extension in the case of an embedded vector field, but it is rather difficult to follow in the case of the feature map. Specifically, Proposition 1 is trivially proved for an embedded vector field, but the same argument should also apply to the case of a feature-map-induced vector field. It would be helpful if the discussion on $g_{w,b}$ could e.g. focus on the case of axis-aligned planes (i.e. each $w$ should be an element of a standard basis and $b = 0$), such that the differential $D_x f$ reduces to an identity. We found this to be significantly more intuitive.
In the general manifold case (Appendix B.5), the idea of projection on pseudo-hyperplane is, although well explained, not at all investigated. First, it is not clear how to practically implement such projections of these pseudo-hyperplanes (not in the geodesically complete case, and even worse in the general case). Second, there are no experiments regarding general manifolds, and there is also no dissertation on the increased computational complexity with respect to hyperbolic and PSD cases. This reduces “contribution 3” in the statement in lines 72-76.
*Minor:*
* The related work section argues that the proposed construction is different from a neural ODE (which also generalizes ResNets), but honestly, we found this argument to be incomplete. It does seem like the proposed construction is a neural ODE.
* Line 126, we found it unclear which "specific structure" is being exploited in methods using Frechet means (these averages apply on practically all manifolds).
*Regarding the score:*
We are willing to increase the score if these concerns are appropriately discussed in the rebuttal.
Questions
Please reply to the weaknesses above.
Furthermore:
Major:
* Line 215 and Appendix B4. The function $l_f(x)$ is inconsistent. In the main paper is a map $M \rightarrow T M$ while in the appendix is a map $M \rightarrow (T M)^*. Can you explain the inconsistency?
Minor:
* Line 189 $f_nn$ should be $f$?
* The Log-Euclidean metric amounts to a Euclidean metric in the tangent space at the identity. In that case, does $f$ reduce to being $f: T_I M \rightarrow R^k$? If so, is your construction just a standard Euclidean ResNet in a pre-specified tangent space? That would be good to state explicitly, if so.
* It's quite common to run into numerical instabilities (esp. on hyperbolic manifolds). Is that something you face?
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Limitations
The parting "Limitations" paragraph discusses a relevant assumption. However, we feel that a discussion of the tractability of (projections onto) "pseudo-hyperplanes" (Eq. 17 in the appendix) is lacking.