Content-Style Learning from Unaligned Domains: Identifiability under Unknown Latent Dimensions

### Response 1/2   **[Length of Main result]** Please note that the entire Section 3 is our main result, which is more than one and half pages. Subsection 3.2 uses “Main result: identifiability without latent dimension knowledge” is part of our main results (Theorem 3.3 and Theorem 3.4). We hoped to articulate the importance of this subsection, but this was indeed a little confusing. Nonetheless, we hope to clarify that the main result is the **entire Sec. 3 and its proofs**. The proof of Sec. 3 is a little lengthy and thus we had to put it in the appendix. We have added a more obvious pointer to make sure that interested readers can easily localize the detailed proofs in the appendix. **We have also changed the title of subsection 3.2 to avoid confusion.**   **[Key difference between our approach and low-rank techniques]** Regarding the comment on “low-rank techniques”, please note that low-rank matrix/tensor factorization methods are often used for tackling linear mixture learning models, e.g., $\boldsymbol{x}=\boldsymbol{A}\boldsymbol{z}$, where x is the observed data, A is the mixing system, and z is the latent code. In our case, since we have nonlinear mixtures, i.e., $\boldsymbol{x}=\boldsymbol{f}(\boldsymbol{z})$ where $\boldsymbol{f}(\cdot)$ is a nonlinear mixing system, the low-rank models do not apply.   **[Assumption 3.1(block independence)]** This is an important point. Thanks for bringing this up. Block independence condition in Assumption 3.1 is a mild assumption that states that the style block is independent of the content block. An example of the content and style being block-independent is as follows: Consider the animal face experiments in the manuscript, where the pose of a dog is the content and the breed is the style. The independence of content and style blocks is implied by the fact that any dog breed (e.g., German Shepherd or Chihuahua) can be combined with any pose (front-facing, left-facing, etc). That is, the choice of pose does not affect the choice of the dog breed---nor does the breed affect the pose. This assumption is commonly seen in the literature [1, 2]. Assumption 3.1 is used in the proof of our identifiability Theorems 3.3 and 3.4 for identifying the style information. Hence it can be relaxed if we only want to identify the content information but not the style information. This assumption may be violated if the content and style affect each other. For example, in the above example, if some breed of dog is physically incapable of sitting on a certain pose, the breed and the pose will no longer be independent of each other. The reasoning behind block independence was briefly discussed after Assumption 3.1 in the original manuscript. **We have further added a remark after Theorem 3.3 explaining when it can be relaxed.**

### Response 2/2   **[Why not implement Problem (4)]** This is a valuable comment that merits further clarification. We briefly discussed the reason why Problem (4) (or its regularized version Problem (5)) was not implemented directly, in lines 262–272 and lines 306-316 in the original manuscript. Simply speaking, Problem (4) is cumbersome in terms of optimization. However, Problem (6) can effectively bypass many of Problem (4)’s challenges, with the price of working under slightly more restrictive conditions. To be more specific, implementing Problem (4) would require several complex elements: independence regularization, latent distribution matching, and injectivity promotion. Independence regularization typically involves non-trivial approaches like minimax optimization [2]. Likewise, latent distribution matching usually relies on GAN-based minimax optimization [3]. Injectivity, in turn, requires either autoencoder-based reconstruction loss [2] or entropy regularization [4]. In contrast, the reformulation in Problem (6) simplifies the process by only requiring distribution matching via GAN, while independence regularization and invertibility constraints are implicitly addressed, as shown in Theorem 4.1. Therefore, we chose to implement Problem (6) instead of Problem (4). We first presented Problem (4) as it explicitly outlines the conditions that are sufficient for identifiability (i.e., injectivity, distribution matching, and independence). It presents the more general conditions under which content and styles are identifiable. Problem (6) is almost equivalent to Problem (4), but it has the same identifiability under slightly more restrictive conditions (i.e., bijectivity of $\widehat{\boldsymbol{q}}$ and simply connectedness of $\mathcal{C}$ and $\mathcal{S}$). However, Problem (6) is way easier to implement (also see our discussions in Lines 306-316 in the original manuscript). That is, Problem (4) enjoys more general conditions and Problem (6) is a reformulation that we made for efficient implementation---both have their own values to be presented. **We have revised Sec. 4 in the new version to articulate and highlight the above points; please see Remark 4.1.**   **[Quoting an optimization problem]** We appreciate the reviewer’s suggestion to distinguish between optimization problems and other equations. **Accordingly, we have revised the manuscript to use “Problem (X)” specifically when referring to optimization problems.**   ### References *[1] Eastwood, Cian, et al. "Self-supervised disentanglement by leveraging structure in data augmentations." arXiv preprint arXiv:2311.08815 (2023).* *[2] Lyu, Qi, et al. "Understanding latent correlation-based multiview learning and self-supervision: An identifiability perspective." arXiv preprint arXiv:2106.07115 (2021).* *[2] Goodfellow, Ian, et al. "Generative adversarial nets." Advances in neural information processing systems 27 (2014).*

### Response 1/1   **[Content features with small norm]** This is an interesting point, and in fact it is a situation that our formulation can avoid. Sparsity regularization results in identifiability regardless of the size of the norm or sparsity of the content features. Let us explain. The proof of Theorem 3.4 sheds light onto the reason why no misidentification occurs. In summary, the sparsity regularization ensures that the learned style cannot contain any content information even when the sparsity dimension is over-estimated. The distribution matching constraint ensures that the learned content cannot contain any style information. Finally, the injectivity of $\boldsymbol{f}$ ensures that all the latent information is preserved among the learned content and style. Hence, the learned content and style contains exactly the content and the style, respectively, leading to identifiability. The fact that the size of the norm of the content variables does not affect identifiability is also reflected by the statement of Theorem 3.4. Specifically, no extra assumption on content’s norm is needed in order to ensure identifiability. This means that the proposed formulation ensures identifiability regardless of the size of the norm of the content features. In our original manuscript, the paragraph in Section 3.2 Line 242 to Line 249 explains the insights regarding the sparsity regularization. **We have also added the above point after Theorem 3.4 in the revised manuscript for clarification.**   **[Computational Efficiency]** Indeed, computational efficiency is an important consideration. Please note that Table 2 has the training time of our method. **Following the reviewer’s comments, we have also added a column of training time for the generation methods.** Basically, the training time of the proposed method is similar to those of the baselines. This is because sparsity regularization, and content and style encoder optimization do not take significant compute resources.   **[Sparsity Regularization for training GAN?]** The need for sparsity regularization is based on our formulation in (4) and the proof in Theorem 3.4. The $\ell_1$ norm is used to approximate the cardinality minimization term in constraint (6b). It is the critical part to ensure that content does not leak into the learned style representations, when the latent dimensions are over-estimated. Please see our discussions in Line 242 - line 249 in the original manuscript. As shown in our experiments (see Figs. 2(b) and 2(c)), if this regularization (i.e., $\lambda=0$) is not used, we observed that the extracted style information could contain a mix of both content and style.   **[References]** Thanks for providing these references. **We have discussed them in the “related works” part.**

### Response 1/1   **[Extension to discrete data modalities]** Indeed, our theoretical analysis is based on the continuous domain $ \mathcal{C}, \mathcal{S},$ and $\mathcal{X}$. Moreover, as rightly noted by the reviewer, our GAN based implementation might not be suitable for discrete and sequential data modalities such as text. Therefore, extending our work to discrete modalities would require extension of our theoretical analysis. The implementation also has to be re-designed. These challenges present interesting future work. We have mentioned them under “Limitations” on Sec. 7 in the revised version, which is as follows: *“Additionally, our work is also limited to continuous data modalities like images, audio, etc. Discrete data modalities like text will require extension of both theory and implementation. This challenge presents another important future work.”*   **[GAN Optimization Instabilities]** The reviewer is correct: GAN-based implementations can be sensitive to small changes in hyperparameters as well as randomness in initialization. In our experiments, we did not quite observe instability due to random initialization. However, we did observe convergence issues when the number of layers of content and style encoders were large (>5). We used only 3 layer fully connected neural networks for both content and style encoders to overcome this issue. In the revised version, we have included the above discussion in Section G.7 “Regarding GAN Stability”. We have also added on remark under “Limitations” on Sec. 7 in the revised version, which is as follows: *“Finally, another limitation of our work is that the proposed method is based on a GAN based framework which is known to be unstable during training. Therefore, novel implementation methods based on more stable distribution matching modules such as flow matching are also of interest as a future work.”*

### Response 1/1   **[Bijectivity of q and f]** This is a nuanced point that is indeed worth clarifying. The bijectivity of f and the injectivity of f are actually **not inconsistent**. The reason is that the bijectivity/injectivity are defined w.r.t. domain and codomain. Let us explain. Note that we assume $\boldsymbol{q}$ to be bijective from $\widehat{\mathcal{C}} \times \widehat{\mathcal{S}}$ to $\mathcal{X}$. Therefore, in Theorem 4.1 (b), our assumption that $\boldsymbol{q}: \widehat{\mathcal{C}} \times \widehat{\mathcal{S}} \to \mathcal{X}$ is bijective implies that **$\boldsymbol{f}= \boldsymbol{q}^{-1}$, when defined w.r.t. $\mathcal{X} \to \widehat{\mathcal{C}} \times \widehat{\mathcal{S}}$ is bijective**. However, this is not inconsistent with the fact that **$\boldsymbol{f}$ is only injective when defined w.r.t. $\mathcal{X} \to \mathbb{R}^{\widehat{d}\_{\rm C} + \widehat{d}\_{\rm S}}$** since $\widehat{\mathcal{C}} \times \widehat{\mathcal{S}} \subset \mathbb{R}^{\widehat{d}\_{\rm C} + \widehat{d}\_{\rm S}}$. Nonetheless, as we mentioned, this is indeed quite nuanced and we take the responsibility of not articulating it more. We have revised the manuscript to clarify this point and given a pointer to Section A.2; see footnote on Theorem 4.1. We have also added the following fact to Sec. A.2 to clarify this point. *“5. If $\boldsymbol{f}: \mathcal{X} \to \mathcal{Y}$ is bijective, then $\boldsymbol{f}: \mathcal{X} \to \mathcal{Z}$ is injective, where $\mathcal{Y} \subset \mathcal{Z}$, $\mathcal{Y} \neq \mathcal{Z}$.”*   **[Relation between different spaces $\mathcal{C}, \mathcal{S}, \mathcal{X}, \mathbb{R}^{\widehat{d}\_{\rm C} + \widehat{d}\_{\rm S}}$]** The relationship between these spaces is indeed also nuanced and crucial for deeper understanding of the work. Let us explain: $\mathcal{C}$ and $\mathcal{S}$ are open sets in $\mathbb{R}^{d_{\rm C}}$ and $\mathbb{R}^{d_{\rm S}}$, respectively. Hence, $\mathcal{X} = \boldsymbol{g}(\mathcal{C} \times \mathcal{S})$ is a $(d_{\rm C} + d_{\rm S})$-dimensional manifold inside $\mathbb{R}^d$. Now, in Theorem 4.1 (b), by assuming $\boldsymbol{q}: \widehat{\mathcal{C}} \times \widehat{\mathcal{S}} \to \mathcal{X}$ to be bijective, we are effectively assuming that $\widehat{\mathcal{C}} \times \widehat{\mathcal{S}}$ is a ($d_{\rm C} + d_{\rm S}$)-dimensional manifold inside $\mathbb{R}^{\widehat{d}\_{\rm C} + \widehat{d}\_{\rm S}}$. This is because $\widehat{\mathcal{C}} \times \widehat{\mathcal{S}} = \boldsymbol{q}^{-1} \circ \boldsymbol{g} (\mathcal{C} \times \mathcal{S})$ and $\boldsymbol{q}^{-1} \circ \boldsymbol{g}$, a composition of two bijections, is a bijection. Hence, although the ambient space $\mathbb{R}^{\widehat{d}\_{\rm C} + \widehat{d}\_{\rm S}}$ is higher dimensional compared to $\mathcal{C} \times \mathcal{S}$, the spaces $\widehat{\mathcal{C}} \times \widehat{\mathcal{S}} $, $\mathcal{C} \times \mathcal{S}$, and $\mathcal{X}$ have the same manifold dimensions of $d_{\rm C} + d_{\rm S}$. Hence one can define bijective functions from one space to another. **We have added a subsection in the proof of Theorem 4.2 to explain the above relationship.**   **[Citation for Proposition E.1]** We double checked and confirmed that the citation of Proposition E.1 is correct, but it is indeed easy to miss---as it is deeply ‘’hidden’’ in the appendix of [Zimmermann et al. 2021]. In the arxiv paper link provided by the reviewer (https://arxiv.org/pdf/2102.08850), please see page 20 section A.5. Effects of the Uniformity Loss, Proposition 5. **We have added this exact location when referencing this work, to avoid confusion.**   **[Consistency of the Notation]** There might have been some misunderstanding. In the paper, $N$ is consistently used as an integer that represents the number of domains. We did not use it for the dimension of the style variables (which is $d_S$). These notations were defined in line 105 and 109 in the original manuscript. This is also the case in Assumption 3.1, where $ \\{ \boldsymbol{s}^{(n)} \\}_{n=1}^N$ represents the collection of style variables of the $N$ domains. We have revised the wording of Assumption 3.1 to make it clearer that each of $\boldsymbol{s}^{(n)}$ itself is a $\mathbb{R}^{d\_{\rm S}}$ dimensional vector, and $N$ represents the number of domains, in order to avoid confusion.

### Response 1/2 &nbsp; **[L1 regularization and sparsity]** The reviewer has a good point: $\ell_1$ is not the only option for approximating the cardinality constraint in our learning criterion. There are many other options that our framework can easily incorporate. For example, the $\ell_p$ function with $p<1$ is often considered a better sparsity promotor than $\ell_1$ (with a higher risk of running into numerical issues due to its nonconvexity). Our framework can also easily incorporate these functions. **We have included experiments using the $\ell_p$ function ($p=0.5$) in the revised version in Appendix H.4.** One can see that the $\ell_p$ function can also alleviate the content leakage problem, performing similarly as the $\ell_1$ norm did. Our take is that, as long as the regularization term is a reasonable sparsity promoter (e.g. the $\ell_1$ and $\ell_p$ functions), the content leakage problem can be largely alleviated---and our framework is flexible to accommodate different choices. **We have added a remark in Section 6 to articulate this point.** &nbsp; **[$\ell_1$ regularization, domain variability, and Injectivity]** We hope to clarify that the $\ell_1$ regularization is not used to justify the domain variability assumption. That the domain variability assumption holds means that the distributions of the styles from different domains are sufficiently different. This assumption does not pertain to the $\ell_1$ or the cardinality regularization. The cardinality regularization is proposed to handle the challenge that the dimensions of the content and style variables are unknown and over-estimated in practice. Similarly, the $\ell_1$ regularization is not employed in order to justify the injectivity property of the mapping function. Even when the $\ell_1$ regularization is not employed, injectivity of the learned functions can still hold. However, the sparsity constraint on the learned style components is critical for separating style and content when their dimensions are over-estimated. For example, let $d_{\rm C} = d_{\rm S} = 1$. Let $ \boldsymbol{x} = \boldsymbol{g}(c,s) = [c, s, 0]^T$. Let $\widehat{d}\_{\rm C} = \widehat{d}\_{\rm S} = 2$. Then $ \boldsymbol{f}\_{\rm S}(\boldsymbol{x}) = [x_1, x_2] $ and $\boldsymbol{f}\_{\rm C}(\boldsymbol{x}) = [0, 0]$ means that $\boldsymbol{f}$ is an injective function and a solution to Problem (4). **But this trivial solution does not separate the content and style**, as explained in Section 3.2 line 244 to line 249 in the original manuscript. Hence the sparsity regularization is designed for ruling out such trivial solutions by “minimizing the information” that can be captured by $\boldsymbol{f}_{\rm S}$. &nbsp; **[Invertibility Assumptions]** The invertibility assumption is indeed quite important. This assumption basically says that **every data sample has an associated unique representation in a latent domain.** It is a hypothesis that is widely used in generative model learning in ML/AI. For example, flow-based generative models [1], (latent) diffusion based generative models [7], and (variational) auto-encoder-based generative models [2,6] all implicitly or explicitly assume such invertibility. The validity of this hypothesis is supported by the practical successes of these models. Most of the latent component identification works adopt this assumption; see [3-6]. We have added a comment in Section 2 in the revised version to reflect the above discussion. Thanks for bringing this up. &nbsp; ### References *[1] Dinh, Laurent, Jascha Sohl-Dickstein, and Samy Bengio. "Density estimation using real NVP." arXiv preprint arXiv:1605.08803 (2016).* *[2] Vincent, Pascal, et al. "Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion." Journal of machine learning research 11.12 (2010).* *[3] Hyvärinen, Aapo, and Petteri Pajunen. "Nonlinear independent component analysis: Existence and uniqueness results." Neural networks 12.3 (1999): 429-439.* *[4] Hyvarinen, Aapo, Hiroaki Sasaki, and Richard Turner. "Nonlinear ICA using auxiliary variables and generalized contrastive learning." The 22nd International Conference on Artificial Intelligence and Statistics. PMLR, 2019.* *[5] Von Kügelgen, Julius, et al. "Self-supervised learning with data augmentations provably isolates content from style." Advances in neural information processing systems 34 (2021): 16451-16467.* *[6] Khemakhem, Ilyes, et al. "Variational autoencoders and nonlinear ICA: A unifying framework." International conference on artificial intelligence and statistics. PMLR, 2020.* *[7] Albergo, Michael S., Nicholas M. Boffi, and Eric Vanden-Eijnden. "Stochastic interpolants: A unifying framework for flows and diffusions." arXiv preprint arXiv:2303.08797 (2023).*

### Response 2/2 &nbsp; **[Experiments on rotated MNIST, where content is not the pose]** The reviewer has made an important and valid point. In the datasets we considered in the original manuscript the content information was the pose. It is of interest to see whether the proposed approach can deal with situations where the content is not the pose. We appreciate the reviewer’s suggestion to use simpler datasets, such as rotated MNIST for clear demonstration. To that end, we have added new experiments on MNIST digits. The first domain is generated by applying a random color to the MNIST digits, whereas the second domain is generated by applying a random rotation uniformly sampled from [-70, 70] degrees to the digits. **The experiment settings and results are detailed in Sec. H.1 in the revised manuscript.** To summarize the results, we see that the proposed method correctly identifies the content (digit identities) and the style (color in domain 1 and rotation in domain 2) parts. Moreover, as suggested by our theory, the sparsity regularization helps to resolve the issue of content-leakage into the learned style part (see Figures in Sec. H.1 in the revised version). &nbsp; **[Minor Remarks]** Yes, it should be $\mathcal{Y}$ instead of $\mathcal{X}$. Thank you for your careful reading and spotting the typo. We have corrected this in the revised version. We appreciate the reviewer’s suggestion to reconsider the terms DM and LDM. We will give it a thought to see if we could find more distinguishing acronyms. Some candidates include probability density alignment (DSA) and latent distribution density alignment (LDSA). We will try to include such considerations in the final version (we have not changed DM/LDM in the current version, as it may need some meticulous work to spot everyone and change them).