Latent Space Translation via Semantic Alignment

Summary

This work proposed a method to translate learned representations between two pre-trained networks, using surprisingly simple transformations. The method is demonstrated on models with different architectures, trained on different modalities and across different tasks.

Strengths

1. The paper is well written and easy to follow. 2. To my understanding, the discussed setting where models are ad-hoc zero-shot stitched, as opposed to being trained to have compatible representations is original. This distinction is important as it makes the approach applicable for the myriad of pre-trained models that already exist. 3. The method is surprisingly simple and yet produces very convincing results as demonstrated through sufficient evaluation.

Weaknesses

1. The exact setting leading to Figure 2 is not clear. Caption mentions results being produced on “CIFAR-100”. However, I was not able to find the entire setting. What two models were used? Are both trained on CIFAR-100? What are the architectures used? Are reported results averaged across multiple settings? 2. Also regarding Figure 2 - the two trends not being monotone is surprising to me. Especially the very clear outliers. I suppose this behavior might occur if each point represents a single experiment and some randomness might affect performance. In that case, it would probably be better to average these results across several experiments as done in other tables in the paper. Can the authors explain this behavior? Minor comments: 1. Typo line 49 - mphasizing -> emphasizing. 2. Notation line 139 - $\mu$ is used in equation 3 but $\bar{x}$ in text. 3. Clarity line 167 - I suggest the “naive absolute baseline” should be concisely explained.

Questions

See questions mentioned in the Weaknesses section.

Rating

8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

4 excellent

Contribution

4 excellent

Limitations

Yes.

Summary

In this paper the authors propose to translate latent space using an effectively angle-preserving affine transformation learned using pairs as anchors. Experiments are conducted on a wide range of tasks showing the okay performance by the swap-in latent space of embeddings/features.

Strengths

**Strengths** The authors have taken an extensive suite of experiments encompassing a diverse set of tasks, including cross-training, cross-architectures, and several downstream tasks. The experimental design demonstrates a comprehensive approach in assessing the proposed method *per se*. Furthremore, this work shows that aligning two latent spaces with zero-shot stitching is poissible without any retraining. This is an important observation and the proposed method could benefit a wide range of downstream tasks.

Weaknesses

**Weaknesses** The following concerns regarding the design of the transformation and its intrinsic limitation warrant consideration. - **Assumption of Linearity and Lack of Analysis in Generative Model**: The proposed method is effectively an angle-preserving affine transformation, which fundamentally assumes linearity of mapping between two latent spaces (of embeddings and/or features). While "linearity of the transformation between seemingly different latent spaces has a theoretical foundation in research on identifiability in neural models," as the author pointed out in the rebuttal, in the context of generative model, the efficacy is still likely contingent on the extent to which two latent spaces exhibit linear mapping, because a good generative model may fill the slight mismatch with its own capacity. In detail, it is likely that an affine transformation may yield only a coarse mapping between two latent spaces, and may not capture fine-grained relations (consider style transfer in generative models). Thus, the good performance in classification tasks may thus be attributed to the capacities of the downstream models rather than the transformation itself. This is a critical assumption that necessitates further scrutiny, but the submission does not provide an in-depth analysis of this assumption. In the current form of the manuscript, for this regard only an autoencoding on MNIST/Fashion MINIST/CIFAR is shown. It does not have a qualitative comparison with the reconstruction from VAE itself, nor quantitative/qualitative results with other stitching and zero-shot methods. The lack of such comparison can unfortunately weaken the justification of the proposed method. Adding such comparison is thus suggested. Optionally, conducting such comparisons on common image datasets of higher resolution, like CelebA, ImageNet, (with more powerful VAE models) may provide more insight in this regard, since it may potentially highlight the capacity of VAE models and the quality of latent space alignment. (**After author response**: See the discussion below) - **Ambiguity in Technical Contribution**: The technical novelty and significance of the proposed method remain unclear. The method effectively is an angle-preserving affine transformation. While this per se is not necessarily an issue, the missing studies mentioned above make it hard to assess the impact and implications of the proposed method. (**After author response**: This is not a major issue anymore, see the discussion below) - ** Position of Works within the Broad Field**: Latent space matching has been the subject of extensive research, albeit in varying contexts such as text, image processing, and generative models. A few (it's even not an attempt to cover the overall picture of this widely studied topic) works below in References that share technical similarity with the proposed method are cited in the References. However, the submission does not include a substantive comparison or discussion with a wide body of works in this field, mkaing it challenging to ascertain the relative standing and uniqueness of the proposed method. (**After author response**: This is not a major issue anymore, see the discussion below) **References**: - Set Prediction in the Latent Space https://proceedings.neurips.cc/paper/2021/hash/d61e9e58ae1058322bc169943b39f1d8-Abstract.html - Formality Style Transfer with Shared Latent Space https://aclanthology.org/2020.coling-main.203.pdf - Homomorphic Latent Space Interpolation for Unpaired Image-To-Image Translation https://openaccess.thecvf.com/content_CVPR_2019/html/Chen_Homomorphic_Latent_Space_Interpolation_for_Unpaired_Image-To-Image_Translation_CVPR_2019_paper.html **Updates** - The previous version of this review mistakenly refer to the page limit. As the authors and the AC pointed out, this is NOT the case and has been corrected. - In Weakness, update the Assumption of Linearity to focus on the generative model. - Added reference to discussion below. - The rating has been increased.

Questions

N/A

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Yes

Summary

This article proposes a method of latent space translation through semantic alignment, using the similarity of latent spaces learned by different neural models on semantically similar data. The method allows for direct conversion of learned representations between different pre-trained networks, and achieve zero-shot stitching of encoders and decoders without additional training. The method is widely validated in various experimental settings, tasks (classification and generation), and modalities, demonstrating the ability to zero-shot stitch neural models under different architecture and modality changes.

Strengths

1. The paper proposes a method for the zero-shot stitching between encoders and decoders on various tasks, enabling seamless integration without the need for additional training. 2. Compared to previous methods, this article expands to different latent space dimensions, architectures, and modalities. 3. Extensive set of experiments have been done to show the possibility to zero-shot together neural models across different architectural and modalities.

Weaknesses

1. In the experiment, only the most basic abs method is compared, but no comparison is made with other methods. 2. In the generation task, zero-shot stitching is performed under the same model framework and training data. Whether it is more practical to migrate between different model frameworks and training data?

Questions

1. How much performance difference is there between the proposed zero-shot stitch method and concatenation followed by finetuning? 2. How is the selection of anchor points determined, and how much impact does different selection have on performance? 3. In experiments, zero-shot stitch is performed between different frameworks on the same dataset. So, can this method be effectively used for neural frameworks trained on different domain datasets?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

2 fair

Contribution

2 fair

Limitations

Yes

Summary

This paper show the latent space of different pretrained models can be translated between each other with simple transformations by using anchors. By using this method, a variety of encoders architectures are able to be cross-stiched to different classifier heads or within autoencoding models.

Strengths

This paper tackles an interesting problem for how to semantically align different trained latent spaces. The paper is a natural extension to Moschella et. al 2023, going from just relative encoding to full latent space transformations. The results are clear in showing zero-shot stitching is possible without retraining the decoders

Weaknesses

It wasn't clear on my first read of the paper that this work is a direct follow up to Moschella et al. 2023. Without that context, the paper was very difficult to read the first time. After reading Moschella et al., I was better able to follow this proposed work. Multiple details are missing, such as the construction of the anchor set, which are glossed over in this work and caused confusion on the zero-shot nature of this work. Another major missing key detail is that prior work's decoders must work on the relative embedding, this crucial detail was not clear until after multiple read throughs. This improvement should be front and center. I find it surprising that tables 1 and 2 do not report Moschella's results. I realize that the decoders must be trained with the relative embeddings, but the lack of comparison feel likes an obfuscation rather than a highlight of the difference between methods. Overall, I find the clarity of the paper to be a major hinderance.

Questions

None

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

1 poor

Contribution

3 good

Limitations

Yes

Thanks authors.

The extra context is great. The clarity issues seem to be due to my unfamiliarity, I will increase my score to weak accept. I still am a little concerned with the final framing, but that's not a content issue.