Common Response
Dear reviewers and AC,
We sincerely appreciate your efforts spent reviewing our manuscript.
As reviewers highlighted, we believe our work demonstrates a novel idea in text-to-3D generation (9SLV, ZxmY, paN1) with theoretical principle (pJwX, ZxmY, paN1) validated with good qualitative examples (pJwX, paN1).
We appreciate reviewers’ constructive comments on our manuscript. In response, we carefully revised and enhanced the manuscript with following:
- Details in comparison with prior works, e.g., VSD (Appendix B)
- 2D experiments to demonstrate the effectiveness of APFO (Figure 14)
- More qualitative results showing the effectiveness of stage 3 mesh refinement (Figure 13)
These updates are temporarily highlighted in “red” for your convenience.
Furthermore, we collected the common questions that multiple reviewers have asked and responded to each question one-by-one in what follows.
---
**Common response 1. Comparison between VSD and APFO**
First, we emphasize that the optimization objectives of APFO (ours) and VSD (prior work) are different even when used with the same timstep schedule of APFO. In specific, as in Eq. (12), APFO scales the difference of denoiser outputs by time-derivative of noise variance, i.e. $\frac{\dot \sigma(t)}{\sigma(t)}$ (with the predetermined timesteps), which exactly amounts the interval to transport latents to the next noise level. In contrast, VSD uses the (empirically chosen) weighting function $\lambda(t)=\sigma^2(t)$ (with randomly chosen timesteps). The difference in training objectives, $\frac{\dot \sigma(t)}{\sigma(t)}$ vs. $\lambda(t)=\sigma^2(t)$, is crucial; even when VSD is employed with a decreasing noise schedule as like APFO, it cannot achieve the main goal of APFO to approximate probability flow ODE, i.e., to emulate the generative process of the diffusion model so that the optimized image follows the data distribution.
For better explanation, we provide a 2D image generation experiment that compares APFO, standard VSD (i.e., random timestep sampling), and VSD with timestep annealing (linearly decreasing as like APFO). Furthermore, as requested by several reviewers, we show an additional comparison with DreamTime [1], which proposes to sample timesteps according to predetermined weighting functions. The results are shown in Appendix B Figure 14 of the revised manuscript. One can observe that the image generated by APFO shows higher fidelity, while other methods show blurry artifacts. Also, the blurry artifact is not reduced even when using an annealing timestep schedule for VSD or VSD scheduled with DreamTime. Also note that APFO produces a more faithful image even when using 200 optimization steps, while other methods use 500 steps. This is because APFO approximates probability flow ODE, which is a straight path towards data distribution, while VSD traverses over random timesteps that prolongs the optimization. This 2D experiment shows that 1) APFO is not only different from simply taking VSD with timestep annealing, but also presents a more faithful image, and 2) by approximating the probability flow of a pre-trained diffusion model, APFO is able to generate an image with significantly fewer optimization steps.
---
**Common response 2. Regarding loss and gradient norm analysis**
In Figure 6 of our manuscript, we plot losses and gradient norms during optimization to show how APFO enables better 3D optimization than VSD. To convey better implication of this figure, first note that the APFO loss in Eq. (13) is equal to the norm of the time-derivative $dx / dt$, i.e.
$$
L = || \frac{\dot\sigma(t)}{\sigma(t)} (D_\phi (x;\sigma(t)) - D_q(x;\sigma(t)||_2^2 \approx || \nabla_x \log p(x;\sigma(t)) - \nabla_x \log q(x;\sigma(t))||_2^2,
$$
where the latter term is equivalent to the Fisher divergence between data distribution and distribution of images from generators. Thus, the convergence of loss indicates the convergence of distribution with respect to Fisher divergence. Also, note that the gradient norm is given by
$$
|| \partial L / \partial \theta || = L ||\partial {\hat{x}} / \partial \theta||,
$$
where $\partial{\hat{x}} / \partial \theta$ is the gradient norm of the generator. Therefore, the convergence of gradient norm indicates the stability of 3D optimization. On the other hand, VSD shows high variance of the loss and gradient norm during the optimization, which is due to the randomly chosen timesteps. We remark that this can be problematic because, first, the high variance implies inconsistent updating for different views, resulting in inconsistent 3D generation. Second, the high variance of the loss implies that the views are kept updated without convergence, so the contents are shifted during the optimization (e.g., Figure 14).
Reference
[1] Huang, Yukun, et al. "DreamTime: An Improved Optimization Strategy for Text-to-3D Content Creation." arXiv preprint arXiv:2306.12422 (2023).