Thanks for the reply
Thanks for the reply. Please find our response below:
> It is interesting to see that CAT3D can be generalized to various condition and target views. Is this capability attributable to that no positional encoding is used in CAT3D? To my knowledge, video models with positional encoding fail to be extended to inference with arbitrary lengths.
While CAT3D does not use an embedding of time (e.g. positional encoding of the time index for each frame), it does use an embedding of camera pose (via the raymap). Unlike time embeddings where during training you only see a small but finite number of time embeddings, we see a much larger continuous set of pose embeddings which may aid in generalization. In video and language models, one still can get generalization depending on how you structure and interpolate the time embeddings (see e.g. [1, 2]).
[1] Chen, Shouyuan, et al. "Extending context window of large language models via positional interpolation."
[2] Kazemnejad, Amirhossein, et al. "The impact of positional encoding on length generalization in transformers."
> For the shift of the noise scheduler, I think the conclusion of using log(N) to shift the scheduler (N is the number of target images) is not convincing. Because when CAT3D is trained with 5 and 7 target views, it is just evaluated with N=5 in the scheduler. This only confirms that using large N helps multi-view training. The specific relationship between the number of views and the scheduler has not been thoroughly assessed in this paper, leaving the log(N) shifting here somewhat ambiguous. Furthermore, it's important to note that multi-view images should not simply be equated with high-resolution images, given that the overlap among multi-view images can vary greatly and is stochastic.
It is common practice in training video diffusion models to shift the noise schedule based on the number of target frames, to compensate for the amount of redundant information which may exist across pixels. This is similar to what is done when increasing a models spatial resolution. In reality, the amount of redundant information in a video is a function of the amount of camera and scene motion, i.e., how many pixels are similar or the same across frames, but the number of frames serves as a reasonable approximation. In practice, our model is adapted from a single image diffusion model, and therefore modifying the noise schedule from that base model is necessary, since supervising and predicting multiple frames strictly has more redundant information. And indeed, in practice, we found shifting noise (while training and sampling) improves the quality of results. It is true that we don’t dynamically adjust the schedule based on the number of target frames, we just use the same shift of log(5) for both 5 and 7 target frames, but we found that the difference between that shift in practice is small (e.g. the average LPIPS on the in-domain diffusion samples is 0.235 for shifting log(5) vs. 0.240 for shifting log(7)). The precise formula for log(N), while not explicitly defined in prior work, was something we derived approximately from the numerical noise schedule information provided in the video model literature [3,4]. We will add this detail to the paper. Future work on multi-view models may want to instead condition the model on logSNR and use different shifts when training and sampling with different numbers of frames.
[3] Blattmann, Andreas, et al. "Align your latents: High-resolution video synthesis with latent diffusion models."
[4] Blattmann, Andreas, et al. "Stable video diffusion: Scaling latent video diffusion models to large datasets."
> I think that providing multi-view generation from diffusion before Nerf learning is very important to confirm the capacity limit of stablediffusion. Even if the model itself cannot be publicly released, sharing samples—such as 80 views generated from Stable Diffusion before NeRF optimization (in the gallery.html)—would greatly benefit the community. This would enable researchers to reproduce performance based on these multi-view images in their own NeRF optimizations.
Great point, we will include more samples alongside our NeRF results in the project page (and will also need to combine those videos with a serialized form of the camera pose trajectories).