Transferring disentangled representations: bridging the gap between synthetic and real images

Summary

The work is situated in the area of Disentangled Representation Learning. The authors propose a new intervention-based metric (OMES) to assess the degree of disentanglement in different models. They perform extensive and comprehensive experiments validating and comparing OMES with other metrics. They thoroughly discuss the behavior of their metric with respect to other metrics and provide intuitions for the observed behavior. Finally, they use OMES to study transfer in disentangled representations from a source to a target dataset. The authors wish to understand which disentanglement properties remain after the transfer which they measure with OMES (and other metrics).

Strengths

The paper is very well written and well structured. The OMES metric is explained and motivated well. The comparisons to the other metrics are comprehensive and sensible. I especially appreciate the formulation of the main research questions in Section 3.1 which guide the logic of the paper. The conducted experiments are very extensive and I appreciate the additional results in Appendix C.

Weaknesses

I do not see major weaknesses with this paper. I think this is a solid and a mature submission. I have some comments and questions which should be well addressable. I found the results section in 3.3. quite hard to read. Each of the transfer experiments consists of one large block of text which is a bit hard to parse. I encourage the authors to think about restructuring it and introducing more paragraphs to promote readability. Fig.1 left: the ylabel should be renamed to “Disentanglement score” for clarity. Table 2: Please write a more descriptive caption ### Typos: Line 33: tranferring Line 169: Agreeement Line 589: Table 7 not Fig 7

Questions

OMES has one hyperparameter alpha in Eq. 1. As far as I have seen, it has always been set to 0.5. Does alpha need tuning / is 0.5 the optimal choice? What happens if alpha is set to a different value? Does this value depend on the dataset? The authors assess the distance between the source and the target data in order to understand how this distance affects transfer performance. While the distance between two datasets is hard to measure, I felt the use of “distance” was a bit loose here, especially because the first research question explicitly involves the distance: “How well does disentanglement transfer, and how much does it depend on the distance between Source and Target Dataset?” In line 51, the authors write: “We discuss the role of fine-tuning and the need to reason on the distance between Source and Target datasets.” I have not really seen any reasoning on the distance between the Source and Target datasets in the paper and would be interested to hear how the authors think about it. The distance could be measured in FoV or in the pixel space or in the “domain” space, for example. We could have the datasets A, B and C where A has 5 FoVs and consists of real images. Then, B would have 4 FoVs (a subset of A) and consist of synthetic, but realistic looking images. Finally, C would have the same 5 FoVs as A, but would consist of sketches, that is, the image distribution would be very far away from A and B. Which dataset would then be closer to A in terms of “distance”: B or C? And how would we expect the disentanglement measures to transfer? I would think that the trained VAE heavily depends on the image domain and switching real images to sketches would destroy all performance in terms of disentanglement.

Rating

7

Confidence

3

Soundness

4

Presentation

3

Contribution

3

Limitations

I think a paragraph which explicitly discusses the limitations of the approach would be beneficial for the paper.

Summary

This paper conducts an empirical investigation into the problem of transferring disentangled representations from synthetic data to real-world data (syn2syn, syn2real, real2real). They start with well-defined research questions and perform the investigation on the feasibility and effectiveness of transfer learning step by step. Besides the investigation, this paper also proposes an intervention-based metric measuring the quality of factor encoding in the representation while providing information about its structure.

Strengths

1. The proposed metric OMES (Overlap Multiple Encoding Scores) is designed to measure two properties of disentangled representations: modularity and compactness. The proposed metric agrees well with other established metrics and also agrees well with performance metrics. 2. The experiments are extensive and comprehensive, covering four application scenarios (syn2syn, syn2real, real2real) and supported by six datasets. Some empirical insights are also revealed from the experiments. For example, the authors suggest that Explicitness is usually well maintained, while Modularity and Compactness are reduced as we move from synthetic to real. Interestingly, the authors also found out that fine-tuning is always beneficial, which is not an expected behavior to me.

Weaknesses

1. The presentation of the proposed metric has poor readability and is not structured well. The description from line85 to line132 is a bit messed up: for example, the input image pair and the motivation for choosing subsets of latent dimensions at line102 should be described earlier, while it might be more appropriate to move the high-level description to the introduction. I would suggest the authors add a few headlines to the metric introduction at least or rephrase the contents. 2. Could you please consider synthetic datasets composed of more complex transformations, such as Falcol3D and Issac3D [1]? Though the datasets in the paper are very diverse, I feel a bit the transformations are relatively too simple. It is not sure to me if these insights transfer to real-world complex transformations. In particular, is fine-tuning still beneficial? 3. How about vector-based disentanglement methods [2][3]? How do the metrics and experiments generalize to vector-based methods? Do the authors think the insights will also transfer to these vector-based approaches? > [1] Semi-supervised stylegan for disentanglement learning. ICML. 2020. > > [2] Flow Factorized Representation Learning. NeurIPS. 2023. > > [3] Unsupervised Learning of Disentangled Representations from Video. NeurIPS. 2017.

Questions

Please see the weaknesses.

Rating

6

Confidence

5

Soundness

3

Presentation

2

Contribution

2

Limitations

Please refer to the weaknesses. There are no particular limitations of this paper -- there is a common limitation of disentangled representation learning that these approaches are not scalable to large datasets.

Summary

This paper proposes a novel classifier-free metric for quantitatively measuring disentanglement and investigates transferring disentangled representations from a synthetic dataset with ground truth factors of variation (FoV) to a target real-world dataset. The authors introduce OMES, a novel intervention-based metric that evaluates the correlation between representations of two images sharing all FoVs. OMES computes the overlap between different FoVs (Overlap Score) and measures the encoding of each factor within single dimensions (Multiple Encoding Score). Using these metrics, the paper analyzes the properties of disentangled representations and source/target distributions to improve disentanglement transfer. The main contributions of the paper are: - Introduction of a novel classifier-free metric (OMES) for disentanglement evaluation, which reduces hyper-parameter sensitivity. - Extensive empirical study on transferring disentangled representations from source to target datasets, revealing the potential and properties of disentanglement transfer.

Strengths

- The paper is easy to follow. - A simple yet novel classifier-free metric removes the dependency on hyper-parameters and enables reasonable comparison between various configurations and benchmarks. Also, this metric maintains the reasonable assessment of disentanglement compared to the conventional metrics and provides an informative tool for analyzing the each factor of variations. - Thorough empirical analysis provides comprehensive understanding on a novel metric and disentanglement transfer learning.

Weaknesses

- The implications on transferring disentanglement from synthetic datasets to complex real-world datasets are somewhat limited. Although the empirical study indicates that a smaller distance between the source and target datasets and shared FoVs between them are beneficial, these are expected properties of conventional transfer learning. The paper would be strengthened by discussing how to specifically select a proper source dataset for a given target dataset with unknown factors of variation. For example, it would be useful to define and measure the structural similarity between a target dataset and potential source datasets. - The presentation could be improved. At first glance, it feels like two independent topics (metrics and transfer learning) are being introduced, making it hard to understand why a novel metric is needed for transferring disentangled representations. It would help readers if the connections between these two components were more clearly explained.

Questions

- In transferring disentangled representation learning, how much of the improvement comes from transfer learning compared to training from scratch on the target dataset, in terms of disentanglement metrics? This comparison would provide a clearer understanding of the benefits and effectiveness of the transfer learning approach. - Can you provide more analysis on why OMES has relatively low correlation on FactorVAE score in Figure 2? - Most of the investigations are done on VAE-based models. Would it have the same implications with other recent disentangled representation learning approaches [1, 2, 3], which employ more powerful generative models? [1] Yang, et al. "Disdiff: Unsupervised disentanglement of diffusion probabilistic models.", in NeurIPS 23. [2] Lin et al., “Infogan-cr and modelcentrality: Self-supervised model training and selection for disentangling gans”, In ICML 20. [3] Ren et al., “Learning disentangled representation by exploiting pretrained generative models: A contrastive learning view”, in ICLR 21.

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

2

Limitations

- Section 3.3 describes the limitation of transferring disentanglement learning.

Thanks for the response

Thanks for the response! I like the new experiments of Isaac3D and I will keep my original score.

Response to Author Rebuttal

I appreciate the authors' effort in addressing the concerns. Most of the concerns are clarified. Although I still believe that including experiments on other disentangled representation learning methods would provide a more comprehensive understanding, I agree that the authors' transferring method is not limited to specific methods and this is not a crucial concern. However, my concern about the comparison to a model trained from scratch remains unresolved. If I understand correctly, their experiments involve two steps: (1) weakly supervised learning (Ada-GVAE) on the Source Dataset, and (2) fine-tuning the model in an unsupervised manner (which does not require GT FoVs). I was asking for the performance gap between step (1) -> (2) (the proposed method) and step (2) alone (training from scratch on the target dataset) to evaluate the impact of step (1). However, it seems that the authors reported the performance of weakly supervised learning (Ada-GVAE) on the Target Dataset, which does not clarify the effect of transferring from the Source Dataset. Please correct me if I have misunderstanding here.

Response to the rebuttal

I have read the rebuttal, the other reviews and the comments. I thank the authors for their responses and clarifications and I am keeping my original score.