Latent Functional Maps: a spectral framework for representation alignment

Summary

The paper demonstrates the possibility to use the functional map tool on the embedding space of neural networks. The idea is that the embedding space of a neural network is usually lay on a low dimensional manifold, and networks that are trained for the same tasks even with different architecture result in similar manifolds in the embedding spaces. It was suggested in the paper to use the functional map as a tool for representation transfer for various tasks. The authors showed that using the functional maps improves over previous methods on several benchmarks (similarity measures, stitching, and word retrieval).

Strengths

The paper presents a novel and elegant idea to utilize the functional map framework for neural representation transfer. In general, the paper is written clearly. In addition, the existing experiment shows superiority in comparison to previous methods.

Weaknesses

There are several concerns regarding this paper that should be addressed by the authors before upgrading the rank. First, it is unclear why there is a need for least squares for computing the functional maps as the correspondence is known by construction. Why not calculate it directly? It is more accurate and does not require the zoomout post-processing. If the authors believe otherwise, they should add an ablation study showing it and explain why this is the case. Additionally, even if it was decided to calculate it via least squares, why the chosen descriptors were used the way they are - as there is no guarantee for their consistency in the two latent spaces. The authors should explain this important point, and may do an ablation study regarding the appropriate descriptors. Second, In general, I would like to see more experiments (or at least similar experiments on more datasets) as in other papers in the field [1]. Additionally, in the stitching experiment, to build the graph the authors used 3000 points with 300 neighbors per point, in that case calculating geodesic distances is almost similar to calculating Euclidean distances in the latent space, which might impair the descriptors. Moreover, the method was not compared to the other baselines. The retrieval experimental setting is unclear. Small issues: I think there is a mix up when using the notation X, Y, M and N (e.g. see caption of the overview figures). [1] Maiorca, Valentino, et al. "Latent Space Translation via Semantic Alignment." Advances in Neural Information Processing Systems 36 (2024).

Questions

see above sections.

Rating

7

Confidence

5

Soundness

3

Presentation

2

Contribution

3

Limitations

The authors should add the known limitations of the functional maps, such as the setting of full and partial manifolds [2, 3, 4]. Additionally, experiments that would significantly strengthen the claims are those that would show that the method presents the same limitations as those used in functional maps for 3D analysis. E.g an experiment showing the part-to-full manifold limits (e.g. training a network with part of the data that another network was trained on, and showing a slanted-diagonal structure of the functional map). [2] Litany, O., et al.. "Fully spectral partial shape matching." In Computer Graphics Forum, vol. 36, no. 2, pp. 247-258. 2017. [3] Rodolà, E., et al.. "Partial functional correspondence." In Computer graphics forum, vol. 36, no. 1, pp. 222-236. 2017. [4] Bracha, A., et al.. "On Partial Shape Correspondence and Functional Maps." arXiv preprint arXiv:2310.14692 (2023).

Summary

This paper proposes using the functional maps paradigm for comparing and aligning the latent spaces of different neural architectures, possibly trained with different setups: initialization, different datasets, noise, etc. The main contribution of this paper is to view the latent space as a Riemannian manifold, and comparing between such spaces is modeled as a correspondence problem. The authors leverage the functional maps paradigm prominent in geometry processing that effectively uses the spectral geometry of manifolds to obtain a compact and accurate representation of the map. The authors demonstrate their idea on 3 (mostly synthetic) setups: (1.) Using F-Maps as a measure to compare between spaces (Sec 4.3), (2.) Stitching latent spaces (Sec 5.2) and (3.) Retrieval using word embeddings. The experiments demonstrate that the functional paradigm is quite good when the number of anchors (ground truth correspondences) is less making a case for the effective use of spectral geometry in such applications

Strengths

- The core idea is interesting i.e. to view a latent space as a manifold and then use spectral geometry and F-Maps to compare multiple such spaces - I found the experiments making a good proof of concept

Weaknesses

- The submission builds considerably on the ideas introduced in [29] and it is important to highlight and compare the key conceptual differences (aside from just "using" F-Maps) - The success of this approach depends on the availability of good anchors and label-based desriptors. Designing these descriptors for 3D shapes (HKS/WKS/SHOT) was straightforward. It is unclear from the submission how these descriptors can be generalized in the setting of representation spaces? - Figure 2 is misleading if the colorbars are not the same. - I found the baselines used for comparison to be rather limited. Why not compare with [18, 22, 25, 27, 20, 13]? To this aid, it is important to show some reconstruction examples similar to Figure 4 in [29] and compare with it - As a general comment, the experimental section seems very hurried. Figures 3 and 5 need a more descriptive caption explaining what is to be learned there? - It would help to make a figure/experiment showing the functional map matrices. While it is acceptable to digest that "similar objects have similar representations" and that "global distances are preserved", I am not fully convinced that this translates into an exact near-isometry model for latent representations. A good evidence would be to show F-Map matrices in different settings. To this aid, there could be some adjustments in the laplacian and descriptor commutativity constraints in Eq 2 and I find it a bit disappointing to not anything new here.

Questions

- What is the significance of comparing LFmaps + Ortho with only Ortho? - Why is the variance in the Fine Labelling case (4b) more than (4a)?

Rating

5

Confidence

4

Soundness

3

Presentation

2

Contribution

3

Limitations

I do not see any direct negative social impact. Overall: I think this is a submission with some interesting findings and the use of functional maps in this setting is quite refreshing. However, I lean negatively on (1.) The lack of good baselines (2.) Stronger evidence on the applicability of functional maps (3.) Lack of clear communication on: progress over [29] and the experimental section not being fully self-contained.

Summary

The paper presents a new way to compare neural representations: Latent Functional Maps (LFM). The later one is achieved by 1) building symmetric knn graphs 2) calculating Laplace eigenfunctions 3) calculating optimal mapping between them. Applications include: (i) compare different spaces in an interpretable way and measure their intrinsic similarity; (ii) find correspondences between them, both in unsupervised and weakly supervised settings, and (iii) to effectively transfer representations between distinct spaces. Experiments with CNNs and word embeddings are provided. The language is fine and the manuscript is well organized, but details of the proposed method and experiments are hard to understand.

Strengths

In terms of novelty, to the best of my knowledge, this is a first time when a concept of Latent Functional Mapping is introduced to deep learning. In this paper, it is applied for comparing neural representations. Developing new scores for comparison of neural embeddings is still an important research topic and could enjoy plantly of applications. LFM is robust to translations in a directions orthogonal to a separating hyperplane, in opposite to well known CKA. Experiments include modern ViT architectures.

Weaknesses

1. The notion of Functional Maps is applied "as is", without significant modifications. So, from the theoretical perspective, there is no contribution. 2. Many details of the proposed method and experiment are missing, instead it authors refer to original papers, see below. It makes the manuscript hard to read. In my opinion, the paper must be self-contained. Settings of these experiments are not a common knowledge. You can include several small paragraphs introducing them to a reader. Examples: * line 152: and incorporate regularizers for Laplacian and descriptor operator commutativity, as defined in [31]: * line 158: in the other latent space. Once we have solved the optimization problem defined in Equation 2, we refine the resulting functional map C using the algorithm proposed by [24]. * line 235: We test the use of the latent functional map in the task of zero-shot stitching, as defined in [29], * line 185: This functional alignment can be used similarly to the method proposed by [29] to establish a "relative" space where the representational spaces X and Y are aligned.

Questions

1. Is the LFM score symmetric? 2. Is it a metric? 3. What is a descriptor operator? (line 152) 4. What is stitching of latent spaces? 5. Why the method from Moschella et al. is not used for comparison? 6. Why do you use this particular set of functions: "As a set of corresponding functions, we use the geodesic distance functions" 7. line 145, 146: do you have one-to-one correspondence between X and Y ? Moschella, L., Maiorca, V., Fumero, M., Norelli, A., Locatello, F., & Rodolà, E. (2022). Relative representations enable zero-shot latent space communication. arXiv preprint arXiv:2209.15430. **Other:** I think that the proposed idea is intersting, especially becase it gives not only a similarity score, but tools for scrutiny of embeddings. But the paper has issues with clarity (see above).

Rating

6

Confidence

4

Soundness

2

Presentation

2

Contribution

2

Limitations

none

Summary

The paper tackles the problem of modeling relationships between latent spaces learnt by different models. It proposes using functional maps that have been used in 3D vision and graph matching for the purpose. It does so by approximating the latent space structure using a knn graph constructed using anchor points, on which the functional map (LFM) is defined. LFM is used to stitch together encoders and decoders from separate CNNs trained independently, where it is shown to perform significantly better and with lesser variability than previous work, while simultaneously using less anchors than them.

Strengths

Tackles an important, practically useful problem using a novel approach while showing strong, although limited experimental results.

Weaknesses

**Writing:** The writing quality in the paper could be significantly improved in terms of: 1. Including more details about the experiments in the main manuscript (defining the goal of the experiment formally, detailing backbones, optimizers used in training, specifying which models/layers are stitched). An example of such a lack of detail is Sec 5.2, L235. 2. Describing closely related past works [22], [29] in more detail. 3. Clarity and organization of Sec 3, 4.2. **Limited Experiments:** The current experimental evaluation seems very limited to models trained on very small datasets, e.g. only CIFAR-100 for the zero-shot stitching despite the relative efficiency of the method. I would like to see if the method performs well on more realistic datasets with more complex latent spaces like ImageNet, or even smaller ones like CUB200.

Questions

I would like to know if the gains in performance due to the method still hold for more practical netowrks that have larger, more complex latent spaces like ImageNet

Rating

6

Confidence

3

Soundness

3

Presentation

1

Contribution

3

Limitations

The authors have sufficiently addressed limitations

Reply to Rebuttal

I have gone through the rebuttal and other reviews, and it takes care of my concerns on experiments, so I increase my score.

Authors made a nice effort.

The authors' made an effort addressing our concerns regarding the computation of the FM, the geodesic descriptor, and the limited experiments. The explanations are clear and satisfactory. We hope that the motivation for the FM computation and the process of construction geodesic descriptors would be included in paper.