In generative modeling, numerous successful approaches leverage a low-dimensional latent space, e.g., Stable Diffusion models the latent space induced by an encoder and generates images through a paired decoder. Although the selection of the latent space is empirically pivotal, determining the optimal choice and the process of identifying it remain unclear. In this study, we aim to shed light on this under-explored topic by rethinking the latent space from the perspective of model complexity. Our investigation starts with the classic generative adversarial networks (GANs). Inspired by the GAN training objective, we propose a novel "distance" between the latent and data distributions, whose minimization coincides with that of the generator complexity. The minimizer of this distance is characterized as the optimal data-dependent latent that most effectively capitalizes on the generator's capacity. Then, we consider parameterizing such a latent distribution by an encoder network and propose a two-stage training strategy called Decoupled Autoencoder (DAE), where the encoder is only updated in the first stage with an auxiliary decoder and then frozen in the second stage while the actual decoder is being trained. DAE can improve the latent distribution and as a result, improve the generative performance. Our theoretical analyses are corroborated by comprehensive experiments on various models such as VQGAN and Diffusion Transformer, where our modifications yield significant improvements in sample quality with decreased model complexity.
Paper
Similar papers
Peer review
Summary
This work investigates what constitutes a good latent space for generative models, and proposes a new training paradigm for generative models – DAE. Simply put, with DAE generative models are trained as Autoencoder in two stages. First, a relatively weak decoder is employed, whose purpose is to aid the encoder in learning meaningful representations. Second, the weak decoder is replaced with the “actual” decoder and training is continued. The second decoder is the one eventually evaluated as a generative model.
Strengths
1. The paper is very well written, easy and interesting to follow. 2. The questions of what constitutes a good latent space and how is one constructed are often overlooked. Most works follow longstanding paradigms of using predetermined distributions (e.g., Gaussian) or whatever is learned from an autoencoder. Posing these questions and formulating the setting, on its own, is impactful. 3. The proposed method DAE is empirically proven effective, and since it is rather simple to implement, I conjecture it might have a strong impact on future generative modeling works.
Weaknesses
While the mathematical formulation in the paper looks sound to me, I think it does not benefit the paper, and significant portions of it could be moved into the appendix. As the authors acknowledge in the Discussion section, the formulation“ serves mainly illustrative purposes” and is “not proven mathematically”. I don’t see it as an issue, as much of Generative Modeling research (and ML in general) is empiric in nature. The proposed method, DAE, could be introduced as an empirically-supported design, while some of the mathematical formulation could be described to serve as intuition (e.g., Theorem 4.1) . However, dedicating over three pages to it seems excessive to me. In my opinion, most readers would greatly benefit if the experiments in Appendix B were present in the main paper instead of some of the mathematical formulation.
Questions
Could the authors please elaborate on “there are other algorithms to implement the decoupled 2-stage training, and the evaluations in this work are not comprehensive”? Why would the authors not include such evaluations in the paper if they find them relevant?
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
3 good
Presentation
3 good
Contribution
4 excellent
Limitations
Authors addressed limitations. One of them (non comprehensive evaluation) raises questions. I would appreciate a clarification on that.
Summary
This paper presents an approach with theoretical analysis to explore a more suitable latent distribution for generation. For this purpose, this paper proposes a novel distance between the latent and target distributions and tries to minimize it to obtain the optimal data-dependent latent distribution. In practice, a two-stage training strategy called Decoupled Autoencoder (DAE) is introduced to leverage a superior latent distribution to improve the generative performance. The experiments on VQGAN and DiT show the effectiveness of the proposed method.
Strengths
1.The paper is well-structured and easy to follow. 2.The methodology is supported by theoretical analysis.
Weaknesses
1.The proposed methodology is somehow similar to AE-OT-GAN [1], which also learns a latent distribution via an autoencoder and then utilizes the learned latent to train GAN. The authors should explicitly highlight the advantages of their approach over [1]. [1] AE-OT-GAN: Training GANs from Data Specific Latent Distribution. ECCV 2020. 2.Some important related works are missed, particularly the line of works that explore improved latent sampling in GANs, e.g., [1] [2] [3] [4]. In special, AdvLatGAN [2] seeks to adjust and discover a more suitable latent distribution via adversarial learning in both the post-training sampling and the training, and it has already introduced the concept of optimal latent distribution noted $p_z^{op}$ in its theory. [2] Improving Generative Adversarial Networks via Adversarial Learning in Latent Space. NeurIPS 2022. [3] Discriminator optimal transport. NeurIPS 2019. [4] Your gan is secretly an energy-based model and you should use discriminator-driven latent sampling. NeurIPS 2020. 3.The experimental results are insufficient. The comparison basically focuses on the performance gain beyond the vanilla backbones. Authors should compare the other efforts on improving the latent sampling/distribution, e.g., the previously mentioned works.
Questions
1.As discussed in [2], the natural image distribution lie upon multiple disjoint manifolds, which potentially requires a discontinuous latent distribution (Prop. 3.2). Does the proposed method adhere to this requirement? 2.In Fig. 2 (a), actually it is hard to discern noticeable performance improvements in the comparison. 3.I am curious why this method can be applied to diffusion models, for which the Gaussian is the stable distribution for noise addition. If the latent distribution is altered, how does the mechanism of diffusion models still function? Please also see the discussion of weaknesses. If the issues are resolved, I would be inclined to reassess and potentially raise my rating.
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
3 good
Presentation
3 good
Contribution
2 fair
Limitations
Please see above.
Summary
The paper first proposes a new framework to analyze latent spaces in the context of generative models. This framework takes inspiration from prior results about GANs, which allowed interpreting the min-max training objective as computing a distance between distributions to be minimized, to define a similar distance between the latent distribution and the data distribution. From their analysis, they derive a simple two-step training for auto-encoders to learn better latents and reconstruction, in which they first train the encoder with a weak decoder to extract good latents, and then train a larger decoder to get better reconstructions. They perform experiments in a simple toy case, and then with commonly used models such as GANs, VQGAN, and DiTs.
Strengths
- The paper is easy to follow - It proposes a novel view on latent codes, including an explanation of some different properties between SSL and generative latent codes, and a theoretical framework to describe why a powerful encoder/decoder pair can't learn a good latent code. - The proposed practical solution is simple and their experiments show that it can improve performances in a variety of generative settings.
Weaknesses
- While the fresh view on latent codes is interesting, it doesn't provide any theoretical guarantees. As far as I understand, the main conclusion is that in order to obtain a good latent code the encoder and decoder should not be too powerful but is not able to give an indication about how the correct balance. Note that in the context of Variational Autoencoders, a similar conclusion had been reached before: that a too powerful decoder is a good explanation for the phenomenon called *posterior collapse* that describes a state in which the latent code is completely uninformative [1]. It is good that it is formally extended to other auto-encoders, but unsurprising. - Because of this lack of quantification of optimal complexity, it is quite unclear whether the positive results obtained in VQGAN and DiT settings are actually related to the theoretical conclusions or not. It could very well be completely unrelated and just confirmation bias. - Related work present different generative models with loose links to the proposed method, but does not discuss any study about the latent spaces of generative models. In addition to links with posterior collapse in VAEs [1], I would also be very interested to read what the authors have to say about sparsity and disentanglement properties of VAE [2], and PCA directions in GAN space [3] among other things. Moreover, I have to point out that contrary to the paragraph in related works, Mask Autoencoders have been explored for their generative capabilities [4]. --- [1] Fixing a Broken ELBO. ICML 2018. Alexander A. Alemi, Ben Poole, Ian Fischer, Joshua V. Dillon, Rif A. Saurous, Kevin Murphy. [2] Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations. ICML 2019. Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Rätsch, Sylvain Gelly, Bernhard Schölkopf, Olivier Bachem [3] GANSpace: Discovering Interpretable GAN Controls. NeurIPS 2020. Erik Härkönen, Aaron Hertzmann, Jaakko Lehtinen, Sylvain Paris [4] MaskGIT: Masked Generative Image Transformer. CVPR 2022. Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, William T. Freeman
Questions
One minor thing that was not clear to me is Remark 3.5. Could the authors explicit why scaling invariance would be a problem in this context? In any case, despite its limitations, I still find the paper interesting for both its theoretical and practical contributions. I trust the authors will fix the related work section, and my current rating anticipates that the authors will demonstrate their willingness to do so. ----- Post-rebuttal update: Having taken into consideration the rebuttal, discussions and comments from the other reviewers, I updated my score from WA to Accept.
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
2 fair
Contribution
2 fair
Limitations
The authors are very open regarding the limitations of their submission. Potential negative societal impacts of generative models are not. Widely acknowledge ones include amplification of biases, potential misuse for propagating fake information, and concerns about lack of attribution and copyright infringements.
Summary
This paper proposes an asymmetric training scheme for auto-encoders that double as image generator. Based on analytical insights that the decoder should have less capacity than the encoder for the encoder to capture correctly the data distribution, they propose a first training cycle where a strong encoder and a weak decoder are trained jointly. This produces a latent distribution able to better capture the data distribution. Then, in a second stage crucial for the end application, the encoder is frozen and a strong decoder is trained using the "good" latent distribution. Experiments are carried out on faces datasets with VQGAN and class conditional imagenet with a diffusion model and show the proposed training scheme is promising.
Strengths
The strengths of this paper are : - It is a simple method, yet effective. As such, it should be easy to reproduce. The improvements are maybe questionable (no error bars, test against the training set - as is traditional in this domain), but given that they are reported with widely different architecture and on different dataset, there is more confidence than usual that it actually works. - The analysis part is really nice. The linear example in particular sheds some light as to why under a constrained budget, the complexity of the encoder should exceed that of the decoder to properly minimize the projected distance (divergence) to the data distribution.
Weaknesses
The main weakness is that most of the paper is about the analytical part, which is a bit handwavy (acknowledged in the conclusion), and lacks a good structure. The reading could be improved by having a clear outline of the work such that one does not wonder where the text is headed to after each section.
Questions
l.146 the link between the ideal $P_z$ and a low capacity generator is tenuous at best. There is no formal proof of that statement as far as I understand the paper. I can see a structural risk argument (ie, among all generator that equally minimize $D(p_x, p_z)$ chose the lowest complexity one), but it is more a principle and not a formal proof. Correct? l. 190 $D(P_z, P_x)$ will be significantly larger in case of arbitrarily chosen $P_z$. This you don't know. If $P_x$ and $P_z$ have the same topology, then there exists a continuous mapping between the two that can be perfectly approximated by a sufficiently large neural network. In practice, maybe $D$ will be larger, but not necessarily in theory. l. 198 I do not agree that $C(F) = C(G)$ is intuitive. For optimal reconstruction, we have $f = g^{-1}$ (assuming all that is required) and the Lipschitz constant of $g^{-1}$ can be arbitrarily large before that of $g$. For example, with $g(x) = x^2$ and $f(x) = \sqrt{x}$ on $(\epsilon, 1)$, the Lipschitz constant of $f$ can be made arbitrarily large. And vice versa. I think this depends whether it is easier (ie, the mapping has lower Lipschitz) to map from $x$ to $z$ or from $z$ to $x$. But what does it depend on? The size of $z$? l. 312 this is a bit cheating. You should learn an encoder that inverse the (frozen) Gaussian space and evaluate on reconstruction for DCGAN. l 354 Why is the Lipschitz of the encoder lower than that of the decoder for the proposed method? Isn't it completely contradictory to all the story of the paper? A2. Fix notation errors.
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
The conclusion is a list of limitations that is very honest.
Disjointed manifolds
For your answer to Q1, there is also a second part: the latent modeling has to be be compatible with multi-modal distributions. I think this is more crucial than the auto-encoder part which, being continuous, keeps the original topology of the natural images unchanged. But this is clearly not a limitation of the proposed method, it is rather a limitation of some of the existing generative models.
Author Response to Reviewer 2zRJ and 14Q5
We thank Reviewer ssMC for your valuable input in the discussion! We agree with your point that latent modeling should be compatible with multi-modal distributions. As pointed out by Reviewer 2zRJ, in standard VAE literature, if the decoder is too powerful will result in **posterior collapse**, that is the stochastic encoder maps data to the prior Gaussian distribution, which is not uni-modal and not informative. From this perspective, our DAE approach utilizes a relatively weak decoder when learning the latent with the aim to better retain the original manifolds in the latent space. As noted in our original response to Q1, if the data distribution is based on multiple disjoint manifolds, the encoder can map them to separate, disjoint manifolds in the latent space.
Thanks for the lengthy rebuttal! I think it covers well the remarks I had in my review.
Author Response to Reviewer ssMC
Thanks for the feedback! We are glad that we have addressed your concerns.
I thank the authors for their answers that adress the main points of my review: - The rebuttal convincingly discuss their relation to other works. - I am still not entirely convinced that the empirical results are a strong validation of the theory but the proposed links are reasonable. Also, both parts are valuable enough regardless. Ideally, I would still appreciate a clarification regarding my question on Remark 3.5.
Author Response to Reviewer 2zRJ: Explaination of Remark 3.5
Thanks for the feedback! We are glad that we have addressed most of your concerns. We are terribly sorry that we didn't explain Remark 3.5. In the following, we first discuss the more general scaling problem, which is asked by Reviewer ssMC, and then specifically explain the arguments in Remark 3.5. The scaling problem might be easier to understand in the encoder-decoder case. Consider the optimal reconstruction case where $f=g^{-1}$ and let $C(\cdot)$ be the Lipschitz constant. For some $\lambda>1$, notice that $f_\lambda := \lambda\cdot f(x)$ and $g_\lambda:=g(x/\lambda)$ have the same reconstruction error, i.e., $g(f(x))=g_\lambda(f_\lambda(x))$ for any $\lambda\ne 0$. In this case, $C(f_\lambda)$ can be arbitrarily large by increasing $\lambda$. This scaling problem remains for the more general cases where $C(f)$ is not invariant to scaling. To address the scaling problem, we considered the generalized $D^{ae}(\cdot, \cdot)$, where both $C(f)$ and $C(g)$ are considered and we want their sum to be small. In this case, $C(f^*)=C(g^*)=1$ is the optimal choice since $C(f^*) + C(g^*) = C(f^*) + 1/C(f^*)\ge 2$ is minimized when $C(f^*)=1$. Now we go back to Remark 3.5. Let us use $P_f$ to represent $P_z$ and the generator has Lipschitz constant less than c, i.e., $g\in G_c$. If we change $P_f$ to $P_{f_\lambda}$ for some $\lambda>1$, we can correspondingly consider $g_\lambda$, which will result in the same reconstructed distribution, i.e., $P_{g(f)}=P_{g_\lambda(f_\lambda)}$. Notice that $C(g_\lambda) < C(g)$ and is also in $G_c$. Therefore, $D^G(P_{f_\lambda}, P_x)$ is non-increasing with $\lambda$ and the optimal $P_{f_\lambda}$ might require $\lambda\to\infty$, which is ill-posed. For a more rigorous illustration, we refer to our Toy Example 1 in Appendix A.1. Please let us know whether we have sufficiently clarified Remark 3.5.
Thank you for the very prompt and detailed clarification. I now see where it can be an issue. I consider all my comments adressed. I will take into account the new elements along with the updates from the other reviewers to formulate my final assessment, in any case before the end of the discussion period.
Thanks for the rebuttal! It addresses the remarks I raised in my review.
Thanks for the rebuttal.
Thanks for the rebuttal. Most of the concerns have been addressed. However, I would still like to note some points regarding the related works. 1."AdvLatGAN [2] further considered the GAN training process and proposed adding an implicit latent transform before the mapping function to improve the latent from its initial distribution." "... they do not characterize the optimal latent distribution." This does not provide a concise summary of [2]. The more related part in [2] is Section 3.2.1, which optimizes the latent distribution to better match the data distribution. In [2]'s Section 3.2.1, the concept of optimal latent distribution (noted $p_z^{op}(G)$ in its theory) has already been included, and [2] claims its optimization could minimize the distance between the initial and optimal latent distribution. The motivation has already been explored to some extent by these previous works. 2."These methods do not consider using an encoder network to parameterize/estimate it." I understand the methodology difference. Nevertheless, the motivation for discovering an optimized latent distribution is consistent across these endeavors. This line of research should not be missed in the discussion.
Further clarification on related work
Thanks for the reply! We are glad that most of your concerns have been addressed. Below, we further address the additional questions/comments. ## Relationship to AdvLatGAN [2] We apologize that our summary of [2] was not comprehensive enough and some important discussions were not included in our initial response. First, we would like to emphasize the key differences between [2] and our work. To clarify, we stick to our notations and use lower-case $g$ to denote a generator and use upper-case $G$ to denote a family of generators. ### The **concept of optimality** is different * The so-called optimal latent $Z^{op}(g)$ in [2] is **conditioned on a fixed generator $g$** and the key property of $Z^{op}(g)$ is that $G[Z^{op}(g)]=x_r$ (Definition 3.1). In comparison, our optimal latent does not depend on any pre-defined generator and is characterized as the **closest in distribution to the data distribution $P_x$**. Although our definition involves capacity/complexity constraint on the family of generators $G$, the resulting optimal $P_z^*$ that minimizes $D^{G}(P_z, P_x)$ aims to directly reflect properties of $P_x$. * The definition $Z^{op}(g)$ contains **no granularity** and does not offer **quantitative measurement** of how close a latent is to the optimal one. For some cases, e.g., $g(z)\equiv 0$, no latent distribution can satisfy $G[Z^{op}(g)]=X_r$ and $Z^{op}(g)$ can be an empty set. In comparison, our distance-between-distributions characterization $D^G(P_z, P_x)$ provides a quantitative measurement for the goodness of a given latent $P_z$. Therefore, our concept of optimal latent distribution **is not included** in [2]. Although both works discussed "optimal latent", **the definition and scope are fundamentally different**. ### The **parametrization of the optimal latent** is different. * In [2], the latent is modeled by a transformation $z^*$ from the original latent, e.g., standard Gaussian. In the proposed AdvLatGAN, there exist constraints on $z^*(z)$ that it cannot deviate too much from the identity map ($d(z_0, z)\le \epsilon$ in equation 5). In comparison, we proposed to model it by an encoder $f$ mapping from the original data, i.e., $P_z=P_{f(x)}$. We believe our parametrization is more flexible and may better capture the underlying structures of the data. ## Benefits of an Encoder As is iterated in both works, mapping uni-modal Gaussian to multi-modal distributions with disjoint supports is difficult and requires the mapping function to be highly complicated. Therefore, in [2], the ideal $z^*(z_0)$ (equation 5 in [2]) might also be highly complicated and hard to learn. We think this is an **inherent difficulty** when modeling the latent by transforming a pre-defined data-agnostic distribution. On the other hand, using an encoder can be more efficient in recovering the optimal latent. It is advocated in our work that the encoder-decoder structure should be used for generative modeling. On a side note, the sampling process of diffusion models also transforms a data-agnostic distribution to the target (in the same metric space). The inherent complexity is also an issue. However, the sampling process consists of multiple steps, and the complexity requirement can be easier to satisfy. Nonetheless, existing works on improving GAN, such as [2], are related to our work. You are totally correct that "the motivation for discovering an optimized latent distribution consistent across these endeavors". **We will add detailed discussions to them in the revision**. Thanks again for the questions! Hopefully, we have clarified the uniqueness of our work.
Thanks for the clarification.
I appreciate the prompt efforts made to distinguish and highlight the distinctiveness of this work in comparison to previous research. The provided clarification is clear and logical to me. I raise my rating to 6.
Decision
Accept (spotlight)