Neural Latent Geometry Search: Product Manifold Inference via Gromov-Hausdorff-Informed Bayesian Optimization

Summary

This paper addresses the problem of automatically identifying the optimal latent geometry for machine learning models. The neural latent geometry search (NLGS) searches for a latent geometry composed of a product of constant curvature model spaces with minimal query evaluations.The proposed method utilizes the Gromov-Hausdorff distance as a measure to compare and evaluate the distance between different metric spaces. By constructing a graph search space and leveraging Bayesian optimization, the method efficiently searches for the optimal latent geometry.Experimental results on synthetic and real-world datasets demonstrate the effectiveness of the proposed approach in identifying the optimal latent geometry for various machine learning problems.

Strengths

- The authors propose a modeling approach where the latent geometry is represented using the Cartesian product of Riemannian manifolds with constant curvature. - This paper constructs a graph search space where each node represents a latent space product manifold. The inverse GH distance is used as the weight on the edges, and Bayesian optimization is employed to find the minimum within this space. - Appropriate figures are included in the paper, effectively illustrating the concepts and results.

Weaknesses

- There is a concern regarding Section 3.1 regarding the potential for overfitting due to the similarity between the objective function $L_{T,A}(g)$ used to determine the latent geometry $g$ and the loss function in the downstream task. - The paper lacks adequate illustration and explanation for the rationale behind utilizing the Cartesian product of Riemannian manifolds with constant curvature, such as Euclidean space, hyperbolic space, and sphere. It would be beneficial for the authors to provide a more comprehensive insight into the reasons for this choice, including potential advantages like efficient computation or other relevant considerations. - Although I am not an expert in image reconstruction and latent graph inference, it could be beneficial to compare the GH-BO method with existing benchmarks for these tasks in order to evaluate its performance and assess its effectiveness and competitiveness. **Other comments:** It seems that the pdf submitted includes the appendix.

Questions

1. Is it possible to utilize the Gromov-Wasserstein (GW) distance, which is a reformulation of the Gromov-Hausdorff (GH) distance [A], as an alternative for practical computation, given the known computational challenges associated with the GH distance? 2. Why was the diffusion kernel chosen to compute similarity in lines 284-287 instead of using the inverse of the Gromov-Hausdorff distance directly? 3. In the experiment, what does the "Naive BO" method use to be the weight on the fully-connected graph? [A] Mémoli, Facundo. "Gromov–Wasserstein distances and the metric approach to object matching." *Foundations of computational mathematics* 11 (2011): 417-487.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

The authors address the limitations of their work. This work does not have a negative societal impact.

Summary

The goal of this paper is to learn the geometry of underlying given data points after mapping them into latent space. Rather than learning a general (nonparametric) Riemannian manifold from the embedding data, the paper models this manifold as a product of certain prototypical manifolds with constant curvature (Euclidean, spherical, hyperbolic). Perhaps the idea is that a general nonlinear manifold can be stitched together using patches from these three types of manifolds. The paper describes a way to map points on these prototypical manifolds to their tangent spaces and back, i.e., the inverse exponential and the exponential map. This way one can express an embedded data point as elements of manifolds using their Euclidean coordinates. Patching together of prototype manifolds is done using a graph representation. With this geometry, the paper derives a framework to fit a latent manifold to the training data. The loss function is built using the Gromov-Hausdorff distance. This requires isometrically embedding the prototypical manifolds into larger Euclidean spaces, and then utilize Euclidean distances to specific Gromov-Hausdorff distances. The search for optimal graph representation of latent space manifold is conducted through Bayesian optimization. Several experiments on synthetic and real-world datasets are presented in support of this theory.

Strengths

-- Tackles a difficult problem of manifold learning by doing to in the latent space. -- The idea of representing arbitrary manifolds by a patching of some constant curvature pieces is interesting. (However, one has to show mathematically that this construction covers all or most of the interesting scenarios.) -- The use of graph representation to patch together the pieces is also interesting and feasible. -- The experimental results are reasonably demonstrative of the ideas although I have not evaluated them too carefully.

Weaknesses

-- Why is the manifold learning or fitting performed in the latent space and not the original data space. This involves a choice of embedding which could be arbitrary. Perhaps the isometric nature of the embedding allows this arbitrariness to be introduced. -- Is it not possible to provided a simple minded sketch of the overall framework first using basic mathematics? There are too many symbols introduced right away and that distracts from a quick, preliminary understanding of the paper.

Questions

Please see my comment above.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The paper does not discuss any limitation specifically.

Summary

The paper introduces an approach to estimate the best latent geometry structure for a machine learning model, trained on a data distribution and for a specific task. The approach assumes a product manifold structure of the latent space of subspace of constant curvature, being able to model product spaces of euclidean, spherical and hyperbolic subspaces. The search over candidate geometries is performed by constructing a graph search space where each node in the graph represents a candidate product manifold, associated with the model performance. Edges represent the inverse of the Gromov- Hausdorff distance between candidate spaces. The search is then conducted using Bayesian optimization algorithms. Experiments on synthetic and real scenarios demonstrate that assuming a graph space which models the topology and geometry of the spaces (with the right distances on the edges) entail finding the correct minima and being more efficient in the optimization process.

Strengths

- The paper experimental results highlight the importance of correctly modeling topology and geometry of the latent space in the search for the product latent space that better suits the data and the task at hand, outlining the importance of modeling the geometry in embedding spaces. - The paper is well written and clear. - The ideas developed and explained in the paper are relevant to the representation learning and geometric deep learning communities: they could be used for new modeling strategies of latent spaces of NNs, developing constraints, and posing the product space search itself as a learning problem to find the geometry that better represents the data for a given task.

Weaknesses

- The approach seems not very efficient to scale up to real case scenarios due to the requirement of embedding everything isometrically in a space of 6n-n dimensions when hyperbolic components are considered. Also it is not entirely evident how the approach would scale up in real scenarios, when growing the number of components (subspaces) and their dimensions considered. - The experiment primarily shows that modeling the topology and geometry is important to search for the right geometry. While this is a good contribution by itself, in my opinion another great potential outcome of the pape would be interesting in showing how the resulting product manifold approximates the latent space, to understand if this tells us something about the data, especially for real dataset (e.g. does the MNIST latent space is better modeled by an euclidean product manifold or different ones, how many components better approximate it, etc..) - It would be good to add some additional works modeling the latent space of NNs as a product manifold, e.g. : - *Fumero, Marco, et al. "Learning disentangled representations via product manifold projection." International conference on machine learning. PMLR, 2021.* - *Pfau, David, et al. "Disentangling by subspace diffusion." Advances in Neural Information Processing Systems 33 (2020): 17403-17415.* - *Zhang, Sharon, Amit Moscovich, and Amit Singer. "Product manifold learning." International Conference on Artificial Intelligence and Statistics. PMLR, 2021.*

Questions

- Does the method scales well with growing dimensions of the latent space or the requirement to embed everything isometrically for hyperbolic latent space is a bottleneck in scaling up everything? - Can the isometric embedding requirement be relaxed and allow for some bounded distortion in the embedding space? Could this help in reducing the dimensionality constraints on the embedding space? - Is the latent space search always made post hoc (after having trained a model)? Do you think that the model training would benefit for enforcing a given geometry during training as well? - The graph pruning strategy (considering models which differ by a single type of latent space and with similar number of subspaces) also loses something in the latent geometry selection strategy? I.e. would it more beneficial in some cases to have two models that differs of more than one component connected in the graph? - How is the neural network employed to decode the signals in experiment in section 4.1 trained? - Are the experiments performed always on a single vector projected on the different product manifold geometries? Especially for real case scenario would not be better to work with multiple samples or some averaged statistics of the latent space? - For real dataset (e.g. MNIST autoencoder) it is correct to say that the result of the graph search provides the best geometry for the latent space among the one considered? Does this tell something specific about the geometry and topology of the data? What happens if the model is trained from scratch assuming the selected geometry, will the resulting model have better performances? - The diffusion kernels signature is nearly invariant to isometric transformation of the graph: does this bring any problem in the graph search (e.g. there are cases in which it would be useful to distinguish two nodes with isometrically similar signatures)? How many components (eigenvalues, eigenvectors pairs) are used to approximate the kernel?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The limitations are correctly addressed in the manuscript.

Summary

A variety of recent advances have provided evidence that learning in the manifold setting (i.e., assuming that high dimensional data lie along low-dimensional latent manifolds, aka. the manifold hypothesis) is advantageous. However, determining the optimal geometry of the latent space often requires expertise or the use of expensive hyperparameter tuning. The present work addresses the problem of learning the optimal latent space geometry for a variety of latent variable models and downstream tasks as well as provides a formal approach to the topic. To solve the problem, the authors introduce the construction of a novel graph search space as a basis for applying Bayesian optimization. The nodes carry information about different latent space geometries and the edges carry weights from the calculation of the Gromov-Hausdorff distance between them. As an example to highlight the details of the approach, product manifolds are used. Experiments with toy and real data sets complete the work and provide empirical evidence for the functionality of the approach.

Strengths

I greatly appreciate any attempt to incorporate geometrically sensitive analysis into machine learning. From my perspective it is advantageous, to assume that data incorporates structural information and to give models the ability to address them. I really like the careful, clear and independent introduction to the theoretical content; It makes it easy (I would assume even for non-experts) to easily follow the authors' arguments. Further, the content is well organized and, with very few exceptions, has no spelling or grammatical flaws. Empirical evidence is accessed through experiments on toy and real world data and the authors make effort to extend the small-scale experimental setting.

Weaknesses

- W1: The theory is presented in a rather narrow setting of product manifolds and the authors simplify the problem setting (e.g., see Section 3.1) by choosing a rather low dimension (i.e., $d_i=2$) of each of the model spaces and restrict themselves to a (very) limited selection of different possible curvatures $K_i \in \lbrace-1, 0, 1\rbrace$. According to Section B in the appendix, estimation of the Gromov-Hausdorff distance in those cases, i.e., in the class of product manifolds parameterized by a limited number of signatures, already seems laborious. The presented method is very model space specific and seems difficult to generalize to arbitrary manifolds. Overall, this impression contradicts my expectations of an automatically optimal latent geometry recognition when learning any latent space models. However, will there always be the need of expert knowledge in the field of differential geometry to first choose a class of manifolds, then derive a class-specific Gromov-Hausdorff distance formula (if tractable or derive sufficient upper bounds) and finally construct a graph search space? - W2: In fact, the authors are aware of, that the class of product manifolds are far from being as rich as, e.g., the class of arbitrary Riemannian manifolds (cf. l. 112ff.). Is it possible to quantify how well this class of manifolds works as a proxy for arbitrary latent space model classes? In other words, in the case of product manifolds as a model class, how severe are the constraints imposed when learning a latent space model, e.g., in the setting of variable Bayesian inference using a variational auto-encoder model? - W3: Considering the points (i) I have rather a broad knowledge on the subject and (ii) the authors admit that to their knowledge no computational method for the mathematical comparison of product manifolds exists (see l. 58), it is nevertheless questionable that no related work exists. I would cherish the effort to address prior work even it is in the broader sense of latent space geometry or addressed to single properties only, e.g., curvature (see Lubold et al., Identifying the latent space geometry of network models through analysis of curvature). - W4: I'm interested in implementation details, in particular how the presented geometry identification fits into a neural network pipeline of a latent space model, for instance in a variation auto-encoder. A structured description of the methodology in form of a code listing as well as code examples for the purpose of reproducibility would be desirable. However, I would like to reiterate my appreciation for the authors' interest in expanding the field of latent space models from a geometrical point of view. I'm sure that by working on the deficiencies mentioned, the work will gain in value and will deliver a valuable contribution to the community.

Questions

- Q1: In Section 4.1, can you give more insight into the "Ground Truth" product manifold $\mathcal{P}_{T}$? Even if this was chosen arbitrarily, what is its dimension? - Q2: What measure is used for evaluation in the experiments; i.e., what does "Minimum" at the y-axis in all plots refer to? And are there any data normalization's involved since I identify no changes in the scale of the plots. - Q3: I would say I got a good intuition for closed balls of radius one in spaces of curvature zero or one, at least for dimensions lower 3. What's a good intuition for closed balls in spaces of curvature smaller zero? Would be cool to have a small section on this, e.g., as part of the appendix. Especially, because those balls are used for comparison of model spaces (see l. 137ff.). - Minor stuff: - l. 129 check wording. - l. 134 $b_0 \in B$ - l. 145 $d_{GH}(B,B\times B')$? - l. 180 $\mathbb{E}^{6n-6}$ ? - l. 199 $d_{GH}(B_{\mathbb{E}²},B_{\mathbb{H²}})$ twice? - l. 352 "Figure 6._In"

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Limitations are addressed (see e.g. Chapter 5) and mainly concern the focus on a limited class of models, namely product manifolds formed by spaces of constant curvature. From my perspective there are important aspects that need to be discussed additionally, e.g., computational limitations in the case of high dimensional latent spaces, since here the estimation of the Gromov-Hausdorff distance in combination with the application of Bayesian optimization usually causes an enormous computational effort. According to section B.4 in the appendix, restrictions related to estimating the Gromov-Hausdorff distance for the (arguably rather simple) model class under consideration are already implied. In the case of arbitrarily complex Riemannian manifolds, do these difficulties even lead to intractability? No evidence was found that the authors addressed potential negative societal impact. Please make the effort to include this mandatory information in your work.

Summary

In the past few years, there's been an interesting trend in the ML literature where people have proposed non-Euclidean latent space geometries to improve network performance. However, to my knowledge geometries have always been picked on an ad hoc basis or tuned naively as a hyperparameter - the most common being hyperbolic and hyperspherical spaces. This evokes a natural question: is there an efficient way to choose the best latent geometry? The authors formulate this choice as a task on its own: neural latent geometry search (NLGS). The authors proceed to propose their own method for NLGS. The method chooses a latent geometry from the set of products of up to a fixed number $n_\mathcal{P}$ of manifolds, where $n_\mathcal{P}$ is a hyperparameter. At a high level, the authors metrize the set of product manifolds, use this metric to structure a graph over the set, and then use this graph as a search space for Bayesian optimization. - The metric is derived from the Gromov-Hausdorff distance between manifolds. The Gromov-Hausdorff distance isometrically embeds two manifolds into a common space ($6n - 6$ dimensional Euclidean space, for this work) and computes the maximum distance from a point on one manifold to the other manifold. Some effort is involved in actually estimating these distance values. - The authors then create a graph whose vertices are product manifolds and whose edges connect those product manifolds which differ by a single term in the product. The edges are weighted according to the inverse Gromov-Hausdorff distance. - The authors then proceed to use Bayesian optimization on the resulting graph to pick the latent geometry which minimizes the loss for the problem of interest. This Gromov-Hausdorff Bayesian optimization (GH BO) method is tested against a naive baseline and Bayesian optimization on a fully connected graph. It outperforms the two methods on both synthetic and real-world experiments.

Strengths

- NLGS is an interesting and well-motivated problem. Based on the broader literature's interest in latent geometries, NLGS could end up being a very impactful framework for model tuning. - As a result, GH BO might also end up being a common tool in the hyperparameter search toolbox. - The theory and machinery used to develop their Gromov-Hausdorff search space is creative and interesting; this is a very original paper. - The whole paper is well-written and easy to follow. At a glance, even the appendices are a nice read. - Their approach very clearly empirically outperforms the baselines, which are reasonable in my opinion.

Weaknesses

1. The choice of Gromov-Hausdorff distance as the metric strikes me as a bit arbitrary. Sure, it's *a* way to metrize the model spaces, but I do not think the paper very clearly justifies it as being the right way for the purpose of ML model performance. Why should I expect two models whose latent geometries are closer in Gromov-Hausdorff distance to perform more similarly? As a general principle of optimization, there needs to be some relationship between the objective function and the search space. My concern here is that this relationship is embodied by the structure of the graph (which is discussed for 1 paragraph), rather than the Gromov-Hausdorff weights themselves (which are developed over more than 2 pages + Appendix B). - A natural ablation here would be to test BO with the same graph as is used in GH BO algorithm, except with uniform edge weights of 1 instead of the GH weights. This experiment would test whether the GH weights, and not just the choice of edges, are responsible for performance. Thus far the authors have only ablated against a complete graph. 2. The experiments are not at all reproducible. - The authors did not describe their network architectures. The appendix only contains a vague description of the autoencoder architecture without details on the number of layers, weights, filters, etc. Even fewer details are available for the other experiments. - The authors did not provide their code. Since their method is somewhat involved, this will also hamper adoption by future work. 3. Clearly GH BO performs better than other methods for the models given, but it's not clear whether these are actually *decent models*. As a result, it's unclear to me how much GH BO will help at scale in realistic settings. Clarifying the model architectures used in the experiments might help. 4. On the theoretical side, several simplifications are made (mostly for computional purposes) which make the final algorithm much less interesting: - The search space is restricted to products between manifolds of dimension 2. - Only unitary (-1, 1) curvature values are used for the non-Euclidean model spaces due to the discrete setup, whereas other works tune the curvature through gradient descent (perhaps that can be done post hoc, but it would be nicer if it were integrated into NLGS). - The authors pruned their search graph by removing edges wherein manifold products differ by more than one model space. - As a result, they only needed to compute 3 distinct Gromov-Hausdorff distance values, resulting in only 3 distinct graph weights. I'm open to raising my score if the authors satisfactorily address concerns #1 and #2.

Questions

What is your opinion on my concern #3? Do you think there's reason to expect GH BO will perform at scale? Can you please clarify this statement regarding other benchmarks: "Furthermore, as there are no existing algorithms to compute the distance between arbitrary product manifolds, we have not included these methods in our benchmark as they would simply converge to random search." - Can you not metrize arbitrary product manifolds using distances within the graph structure you've designed? - Why would they converge to random search?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The authors identify *constant curvature* model spaces as a limitation. They also point to limits in the field of Bayesian optimization on graphs. However, I believe some of the simplifications I pointed out in the weaknesses section are also pertinent. It would be helpful to either (1) explicitly raise these as limitations or (2) further justify them as good choices for the algorithm.

Thank you for the rebuttal

Thank you for addressing my questions and concerns. It's cool to see the GH edge weights substantially outperforming unit weights on the first task. Empirically at least, it appears that some amount of *smoothness* (thank you for defining this) applies to the GH search space. As promised, I'll raise my score, conditional on including these results and the architectural details in the final paper, and on releasing the code. I ask, however, that you also include the graph BO baseline in the two experiments in section 4.2.

Response to rebuttal

I thank the authors for answering my questions and concers. - **[Q2]** Thanks for the answer. In practice is still not entirely clear to me how costly it is be to scale the method to higher dimensional model spaces. Can the author elaborate more on the gap to cover to make the approach usable in practice (this is why I was mentioning approximations in my previous comment)? I think this would be useful also concerning future work. - **[Q6]** Thanks for the answer. Concerning real models does this mean that the selection is made at initialization? What about trained models? how the input vector is chosen in that case?

Response to authors

Thank you for your clarifications that effectively address my concerns. Your explanations really help me to understand the manuscript. After thoroughly reviewing all the reviews and rebuttals, I have decided to adjust my score from 4 to 6.