Norm-guided latent space exploration for text-to-image generation

Summary

This paper proposes a novel method for interpolating between two seeds and demonstrates that it defines a new non-Euclidean metric that takes into account a norm-based prior on seeds. This paper describes a simple yet efficient algorithm for approximating this metric and using it to further define centroids in the latent seed space, which helps generate rare concept images and leads to state-of-the-art performance on few-shot and long-tail benchmarks.

Strengths

- This paper first discusses the property of the seed, that is, the relationship between the norm of the seed and the quality of the generated image, which provides good theoretical support for the proposed new non-Euclidean metric. - The newly proposed non-Euclidean metric combined with the centroid method has a good effect according to the experimental results and has been optimized to a certain extent for problems such as rare concept generation and long tail training. - From the seed level, the paper investigates the text-image generation problem of the diffusion model, and verify the feasibility of controlling the image generation from the seed level.

Weaknesses

- In the early stage, the norm problem of seed was elaborated and verified too much, but the core path optimization and centroid method did not elaborate enough. - Judging from the experimental results of the pictures in the article, the method in this paper does not seem to show a particularly great advantage, especially after adding the seed-selection method. - The two interpolation methods compared in this article are very basic. There are more non-linear interpolation methods to compare, and the results after interpolation seem to be inconsistent with the actual results. In my understanding (also my experiment testing), no matter what kind of Gaussian noise the seed is, SDM can generate a relatively reasonable image, rather than an unnatural noisy image like the ones presented in Figure 1(left). - From the point of view of experimental design, the seed select method is also a key part, such as Figure 5, but it has not been explained in detail.

Questions

- There are some formatting errors in the article, such as the picture in the upper right corner of page 4 without icons and annotations. - I can not understand the interpolation path of 2D space. Hope to have more explanations of 2D space, and how to get the result such as in Figure 1 (right). - Other concerns have already been mentioned in Weakness.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Yes, the authors have addressed the limitations and potential negative societal impact of their work.

Summary

The paper observed that the seed (noise) for the trained diffusion model has a property that the norm of the seed, which follows the $\chi$ distribution, is concentrated around a certain positive number, $\sqrt{d}$ where $d$ denotes the seed dimension. Based on this observation, the paper proposed a way to explore the seed space (e.g. interpolation, centroid) using Norm-Aware Optimization (NAO), which defines the objective function as likelihood maximization in the seed space. The paper applied seed space exploration with NAO to generate rare concepts and augment semantic data for few-shot classification and long-tail learning.

Strengths

- The paper clearly stated contributions with comparisons with existing works on latent space exploration. - The proposed method relies on the inherent structure of latent space, which is defined as the normal distribution, so could be used generally under diffusion model literature. - Experiments are well-designed and easy to follow.

Weaknesses

- Some experimental conditions (e.g. number of piece-wise linear paths) are unclear. - Although the proposed method uses multiple approximations, the paper does not provide analysis or experiments on the accuracy of the approximation. - Comparison for the rare-concept generation using centroid estimations seems not fair. Compared baseline is initialized randomly, but the proposed method is initialized with the Euclidean centroid.

Questions

- It is unclear how many piece-wise linear paths are used to estimate (1) in experiments. Also, there is no discussion or analysis on $\delta$. If one set small $\delta$, the objective function (2) would be harder since the number of variables increases. If one sets large $\delta$, the approximation will be inaccurate. The paper mentioned that they constrain consecutive path points to be close (lines 161-162), but the corresponding description is unclear. Especially, in line 212, what is the meaning of the constraints, $c(x)\leq0$? There is no definition of the function $c(\cdot)$ and the input argument $x$. - For the centroid estimation, there are at least two approximation gaps: one from discretizing the line-integral of log-likelihood (2) and the other from sub-optimal solution for (2). However, there is no analysis of the accuracy of the proposed distance function and centroid estimation, except for empirical performance whose accuracy seems to be dominated by the Euclidean centroid initialization. - For the rare-concept generation, the paper claims that SeedSelect with the initial point found by NAO-centroid achieves faster and better generation. However, NAO-centroid also initializes its centroid with the Euclidean centroid of inversion points whereas SeedSelect uses random initialization. For a fair comparison, the runtime and performance of the SeedSelect initialized with the Euclidean centroid should be compared. Note that the Euclidean centroid initialization of the NAO-centroid is “to speed up convergence” (in line 201). - In Table 4, CIFAR-FS $T_{Opt}$ might be 21 sec, not 21 min. And there is no unit for miniImageNet $\bar{T}_{Opt}$.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

Yes, the authors describe the limitations of the proposed method.

Summary

This paper investigates a new method for interpolating in the seed space of diffusion models, which is the Gaussian distribution used to initialize the generation process. Experiments demonstrate that diffusion models struggle with generation when the norm of the input differs from the distribution of norms of random noise samples drawn from the starting distribution; this can happen if the input is a LERP or SLERP interpolation between two random samples. The paper proposes to define a prior over the seed space using a chi distribution, and finds a path such that the likelihood of each point on the path is maximized under the prior distribution. This is optimized in a discretized fashion. A similar method can be used to find the centroid between multiple images, which minimizes the likelihood of the paths from each latent to the centroid. These methods are used to generate additional data, and studied in the context of rare-concept generation, few-shot recognition, and long-tail recognition.

Strengths

- The proposed method is demonstrated to generate effective images in limited data scenarios. By providing a few example images, the model can generate related images by interpolating between the inputs under high likelihood regions of the latent space. - The method is effective when combined with prior existing methods. SeedSelect optimizes an initial seed to match the concepts in a few given images. When using the NAO-centroid to initialize the seed, the results are more effective and optimization is faster compared to using SeedSelect alone. - Qualitatively, the results look compelling against other shown interpolations in the diffusion seed space.

Weaknesses

- It seems that the investigation is only performed on the input latent space, but I'm curious if feature interpolation in alternative latent representations would preserver a stronger image prior. For example, Asyrp[1] demonstrates that smooth changes can be obtained by manipulating the h-space of a diffusion model. I think this would be a worthwhile baseline to compare to, rather than just interpolations in the input latent space. - The method relies on optimization over over a set of points. I think more details on the optimization could be provided here -- for example, how do the results differ if the number of interpolation points is changed, or the optimization time changes? What is the variation in this optimization procedure? [1] Asyrp: https://arxiv.org/abs/2210.10960

Questions

- in L212, what are the constraints $c(x) \geq 0$? - Figure 1 left was difficult to understand in the first pass. Perhaps it would help to make clearer that the color refers to log likelihood of the $\chi$ distribution? - Table 4: should it be 21 seconds rather than minutes? - Are the optimization times stated per image in Table 4? - What is the total overall overhead for the long-tail experiments -- how many images are generated per class?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Limitations are addressed in the conclusion. The key limitation is that this method requires additional optimization to generate one image, on the order of 30-60 seconds. The most effective use case of this method seems to be in conjunction with SeedSelect to produce plausible images with less optimization time, as the NAO method alone does not always produce recognizable images, as shown in Table 2.

Summary

This paper makes the observation that current training procedures make diffusion models biased toward inputs with a narrow range of norm values. To address this issue, the authors propose a novel method for interpolating between two seeds and demonstrate that it defines a new non-Euclidean metric that takes into account a norm-based prior on seeds. The authors describe a simple yet efficient algorithm for approximating this metric and use it to further define centroids in the latent seed space. The effectiveness of the proposed approach is validated on generating images of rare concepts, and augmenting semantic data for few-shot classification and long-tail learning.

Strengths

1. The observation that current training procedures make diffusion models biased toward inputs with a narrow range of norm values is interesting and inspiring. 2. The proposed approach is well-motivated and aligns with intuition. 3. The observation and proposed approach have the potential to benefit many tasks related to the application of diffusion models.

Weaknesses

1. The proposed approach only works well with seed optimization techniques such as SeedSelect, as indicated by Fig.5 and lines 195 - 203. This implies that the derived interpolation paths may not be optimal and it also introduces extra computational cost of the seed optimization. 2. The proposed approach is demonstrated mainly on generating rare images and augmenting data for few-shot learning. However, I am more interested in more applications that might benefit from the proposed approach. For example, can the proposed approach be applied for video interpolation and video generation and perform better than previous approaches? 3. Although significantly better than previous approaches, the results in Fig.3 indicate that the interpolation results of the proposed approach is still not perfect. Do the authors have insights on the reason for the imperfect interpolation results?

Questions

Please refer to the weakness section.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The limitation is adequately addressed.

Response to author rebuttal

Thanks to the authors for the response and additional clarifications. I agree with the other reviewers that the baselines presented are relatively simple, but I think the method is promising with regards to tasks in which data is limited by allowing a generator to produce additional samples from the limited data points (few-shot and long-tail applications). However, I find that improving the clarity of the paper would extremely helpful. I have updated my rating accordingly.

Thanks for the response.

I appreciate the response from the authors. I apologize for the instances that I missed during the reviewing of the paper and most of my concerns about the lack of experimental conditions are adequately resolved. Also, I'm pleased to hear that the authors will consider including a discussion about the approximation accuracy. My final question is about the Q5. The reason I thought that the SeedSelect is randomly initialized for *Rare-concept generation* is the lines 255-256. > SeedSelect [43] is a baseline method where a seed is randomly sampled and *no centroid is calculated*. Can authors clarify this? Does *no centroid is calculated* mean Euclidean centroid?

Thanks for the response.

Thanks to the authors for the response and additional clarifications of my questions. I understand that the significance of choosing two basic methods is to explore the distribution of the seed space. In addition, I accept the author's explanation that SD cannot generate visual images usually, and I hope that a brief explanation can be given in the final version. I have updated my rating accordingly.

Thank authors for the rebuttal

Thank the authors for the rebuttal. The authors have addressed most of my concerns so I updated my rating accordingly.