Rethinking Score Distillation as a Bridge Between Image Distributions

Summary

The paper analysis score distillation methods in a single framework and hypotheses about two possible error sources - the single step ODE solver approximation and the mismatch between the assumed and true source image distribution. It then proceeds to tackle the first of them using a custom negative prompt design. The authors show this improves the quality of the generated images in both 2D and 3D tasks as measured by qualitative metrics and in a user study.

Strengths

S1) The analysis of different methods in a single framework and with illustrations and experiments is quite interesting for a reader and useful for understanding the SotA and its motivation. S2) The exposition is very clear except for the treatment of the related work discussion. S3) The proposed negative prompt idea is very effective in the tested scenarios. S4) Quantitative metrics are complemented by a user study. S5) The "3D Sketch-to-Real" task hidden in the Appendix seems quite original but it would require more examination (and probably a separate paper) to become a substantial contribution on its own.

Weaknesses

W1) The technical contribution of the paper is limited. Specifically, I see the DDS gradient [18] and the new proposed gradient (eps_ours) as closely related. As far as I can tell, the formulas are the same and the difference is in the choice of the source prompt. Since DDS is presented to work with arbitrary source and target prompt pairs, I feel the new method could be postulated as a specific realization of DDS for one specific source prompt. The authors also omit DDS from all comparisons which supports this viewpoint. The paper still presents a novel analysis of performance in setups that were not shown in the original DDS paper [18] but I am worried the way the method is presented here ("Ours") is not optimal from the perspective of originality. W2) The idea of discouraging artifacts using a negative prompt is likely not a substantial scientific contribution since negative prompts are commonly used by practitioners in image generating interfaces such as Midjourney. W3) The authors identify two sources of errors - the 1st order approximation and the source distribution match - but they mostly only analyze the latter. The former is speculated to improve performance of ISM but if this hypothesis should be considered seriously, I would expect the multi-step ODE solver to be tested in combination with the various gradient definitions. This could show it in fact helps consistently and how orthogonal it is to the other error type. Currently, not even the basic ISM is included in Table 1 so I find the "First-order approximation error" theory plausible but untested. W4) Other suggestions for the analytical Sec. 2.3: - The analysis of SDS focuses on cases with s >> 1 but it does not explain the poor performance with s = 1. - Unlike for SDS, the analysis and Figure for DDS omits the effect of CFG while the original work uses s > 1 ([18]: Eq 3 and Fig. 5). I would expect to see some of the same distribution shifts as in SDS. W5) The related work is deferred to the appendix. The discussion covers the various distillation methods and their applications quite well but due to its location the paper alone is sometimes a bit hard to follow. Therefore, I am personally not very keen of this design choice. Notice that I do not consider this a reason for rejection but I still put it among the negatives since I see it as a minor weakness. Minor presentation issues and suggestions: - [25] is NeurIPS 2022 - When Fig. 1 is first referenced the meaning of different eps suffixes has not yet been defined in the text. ----------- **Justification of recommendation** Overall this could be a very nice meta-paper on score distillation but to that goal it unfortunately only explores one part of the problem - the score definition - and not the ODE approximation. The negative prompting idea seems effective but not very novel given the common usage of negative prompts in image generation and the similarity of the score gradient to DDS. Therefore I lean towards rejection. ----------- **Post rebuttal** I increase my rating after the rebuttal because the authors could show that their method is useful also in 3D generation and it also can be combined with the multi-step inversion to demonstrate influence of both limiting factors discussed in the theoretical part. These should be integrated into the paper. I keep my scope just positive since I still see the method as technically very incremental (e.g. the main difference from DDS is just the input image) and I await the discussion of the other reviewers.

Questions

Q1) The authors use CFG strength s=40 (L483) or even s=100 (L487) to test their method which is a lot more than the range 3-15 considered in the DDS paper [18]. Why the difference? Q2) The definition of the score function in Eq. 2 differs from [25] and also from what I would expect. It is a bit hard to judge because $\epsilon_t$ is also not at all defined but I would expect $\sigma_t$ to play a role, ie. smt like $\epsilon_t/ \sigma_T^2$ as in [25, Eq. 3]. Can you please clarify this?

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

The authors discuss impact of AI art on society and mention that the performance of their method does not match the reverse sampling.

Summary

This paper proposes interpreting score distillation sampling (SDS), a widely used method for generating 3D, 4D, and vector graphics, through the lens of the Schrödinger Bridge (SB) problem. According to the paper, SDS is a linear approximation of the optimal path moving from the current distribution to the target distribution. The paper identifies two sources of approximation error: first-order approximation error and source distribution mismatch. To address the second error source, the authors suggest a simple method: using a text prompt that describes the source distribution instead of a null prompt. This approach is computationally more efficient than the best variant of SDS, VSD, and the authors demonstrate its effectiveness across various tasks such as text-to-image, text-to-3D, painting-to-real, and illusion generation.

Strengths

- The interpretation that SDS finds the optimal path connecting two distributions is novel and aids in understanding the behavior of the widely used SDS. - The paper addresses the oversaturation problem of conventional SDS and is computationally more efficient than VSD. - The paper demonstrates effectiveness across a wider range of tasks compared to previous papers (SDS, NFSD, and VSD). - The paper is well-written and easy to follow.

Weaknesses

- Up to section 2.3, I enjoyed reading and expected a principled solution. However, the solution in section 2.4 was quite naive and heuristic. A major drawback of this solution is that the pre-trained diffusion model needs to accurately match the proposed descriptions such as "oversaturated, smooth,…" with the source distribution. Considering that the text-to-3D experiments are based on Threestudio, my assumption is that the text-to-image model used in this paper is Stable Diffusion 2-base (the exact model is not mentioned in the paper). **It is questionable whether other models (MVDream, SDXL, PixArt, SD3, etc.) can understand these descriptions well.** My guess is that diffusion models trained on such high-quality data will still generate clean images even when descriptions like "oversaturated, smooth,…" are appended, and therefore will still suffer from the source distribution mismatch problem. - The paper does not consider the **Janus problem**, which frequently occurs in text-to-3D. It is questionable whether the proposed methodology would be effective with MVDream [50], a pre-trained diffusion model that addresses this issue. - **VSD has the strength of** not only ensuring the quality of rendered images but also **achieving sample diversity as the number of particles increases.** Therefore, line 282 is not true, and the proposed method falls behind VSD in terms of sample diversity.

Questions

- What pre-trained diffusion model did you use in this paper? - I personally tried text-to-3D with SDS using Diffusion-DPO [A, B], which is a post-trained diffusion model for aesthetic quality, and failed to generate plausible geometry. Can this phenomenon be explained from the perspective of the Schrödinger Bridge problem? [A] https://huggingface.co/mhdang/dpo-sd1.5-text2image-v1 [B] Wallace et al., Diffusion Model Alignment Using Direct Preference Optimization, CVPR 2024.

Rating

5

Confidence

4

Soundness

2

Presentation

3

Contribution

2

Limitations

The limitations of this paper are stated in section 4, but they need improvement. As mentioned in the weaknesses, unlike the method proposed in this paper, VSD can resolve the diversity issue by increasing the number of particles, so line 282 is misleading. Additionally, it is necessary to mention the slow generation speed of SDS and the failure to address the Janus problem in text-to-3D, for the benefit of the readers.

Summary

This paper revisits the application of Score Distillation Sampling (SDS) for tasks with limited data availability by proposing a new interpretation based on Schrödinger Bridges for optimal-cost transport between distributions. The paper highlights that existing SDS methods produce artifacts due to linear approximations and poor estimates of source distributions. By aligning the text conditioning more closely with the source distribution's characteristics, the authors demonstrate significant improvements in image generation tasks such as text-to-2D and 3D, and translation between art styles. The proposed method avoids the computational overhead of previous approaches while achieving comparable or superior results in terms of image quality across various domains.

Strengths

- Overall I find that the writing is clear, concise, and well-structured, making it easy for readers to follow the arguments and understand the key points. I really like the view of optimal transport between source and target distributions to understand score distillation. - This paper provides a comprehensive analysis of existing methods from a unified point of view. They further propose a simple yet effective method in transferring 2D diffusion prior to the 3D scene generation or editing. In contrast to prior state-of-the-art ProlificDreamer, it does not require fine-tuning of diffusion models, which may introduce training inefficiency and instabilities

Weaknesses

- The description of the proposed method is quite concise. Some technical parts lack enough rationales. For example, estimating the source distribution by negative text prompts lacks the rationales. I believe providing detailed analysis with ablation studies will make the paper more informative to answer the following questions: How do you choose these negative prompts? Does it need to be hand-crafted carefully every time adapting to a new domain/task? Why do you have to propose a two-stage optimization pipeline? - How do you choose the value of w? If the value of w is still very large, then the proposed method likely inherits the weaknesses of using a high CFG with standard SDS: less diversity of generations for a given prompt, less realistic generations, and over saturated colors. Besides, if the value of w is still very large, I think the proposed method is more similar to directly using the original negative prompts as guidance. - The text-to-3D experimental results are somewhat not convincing since the comparison of both qualitative and quantitative results is inadequate and more competitive baselines should be included, such as Fantasia3D, Magic3D, CSD, NFSD, etc.

Questions

See the weakness section for questions. I would like to raise my scores if the above questions are addressed.

Rating

6

Confidence

4

Soundness

3

Presentation

4

Contribution

3

Limitations

Yes.

Thank you!

Thanks for updating your review!! We will incorporate these suggested experiments in our revised version. Regarding DDS, we also wanted to mention that it uses the same negative prompt throughout the optimization, which our new ablation study shows is highly ineffectual. We want to iterate that the contribution of our paper is the proposed framework, which enables the understanding and the proposal of a two-stage optimization pipeline.