Reply to Reviewer MCcD
We thank the reviewer for the reply and respectfully disagree with some of the claims.
## (Q1, Q3) Our model does not have enough ability to generate large-scale camera changes:
**We first argue that our current model can handle large camera motions to some extent, as shown in our results**. For example, sample #3 in the first generation results in our demo video (0:01) are controlled by very different cameras; In the first demo of our 6-view video generation (1:35 in the demo video), the differences between the view on the left-top and the one on the bottom-right are also significant. These examples represent some of the most diverse and extreme camera trajectory variations possible that we can think of, within a 16-frame sequence, demonstrating that our method performs well even when the camera moves in opposite directions (e.g., flying in vs. flying out, rotating left vs. rotating right).
Second, CVD is grounded in the currently limited infrastructures of open-source video generation and available datasets, functioning as a proof-of-concept that maximizes the potential of existing resources. This small-scale setup naturally affects the model’s capability to support large camera motions for two reasons:
The capabilities of our method are inherently constrained by the performance of our base models, namely CameraCtrl. Consequently, the extent of motion and the range of camera paths we can manage are limited by the pretrained AnimateDiff and CameraCtrl models. Since the camera path conditioning is derived from RealEstate10K’s camera paths in CameraCtrl (and other models such as MotionCtrl, VD3D), our method cannot accommodate dramatic camera changes beyond what is covered in the RealEstate10K domain. As an extension of existing camera-conditioned video generation techniques, our approach necessarily inherits these limitations.
At the time of our paper’s development, open-source video generation models were generally limited to producing only a few frames. For example, AnimateDiff, the base model used in our work, is capable of generating only 16-frame videos. Within this frame limit, accommodating large motions and significant camera movements is particularly challenging.
Although CVD is relatively small in scale, we are inspired by our results and see significant potential in a scaled-up version incorporating systems like SORA or DreamMachine. We believe our work will inspire and motivate future research and development efforts in multi-view and multi-camera video generation.
## (Q2) Our model lacks generalization by using different LoRA for different experiments:
We want to clarify that it is a common practice for diffusion models to apply different LoRA in different tasks. Both CameraCtrl and AnimateDiff are trained with specific LoRA to adapt the training data distribution, but are combined with unseen LoRAs in their qualitative results.
While we apply different LoRAs for the spatial attention layers (which are pretrained models from previous works) for each task, our CVSM module remains the same across all experiments. We also want to emphasize that, although during the training our model **has never seen RealVision’s LoRA module**, it can naturally adapt this LoRA during the inference. These indicate that our CVSM can be generalized in various tasks.
On the other hand, even though we have shown that our model generalizes well to unseen LoRAs, this does not mean our method does not work well without a LoRA. To support this, we also provide additional results without using any appearance LoRA (along with results in Table 1. in our paper):
Model w/o LoRA:
Rot. AUC: 21.9 / 34.6 / 49.4
Trans. AUC: 2.7 / 7.4 / 16.3
Prec.: 44.0
M-S: 17.6
Full Model:
Rot. AUC: 25.2 / 40.7 / 57.5
Trans. AUC: 3.7 / 9.6 / 19.9
Prec.: 51.0
M-S: 23.5
MotionCtrl [56]+SVD:
Rot. AUC: 12.2 / 28.2 / 48.0
Trans. AUC: 1.2 / 4.9 / 13.5
Prec.: 23.5
M-S: 12.8
Our model without any LoRA slightly underperforms our full model but still performs much better than the baselines.
We also notice that merging different LoRAs into a single model is an active area of research (e.g. Work from Gu et al. [1] and Po et al. [2]). Therefore, while our setting matches standard practice in the field, we believe it would be interesting to explore CVD in combination with very recent advances, like MoS or orthogonal adaptation, which we will clarify in the discussion.
[1] Mix-of-show: Decentralized low-rank adaptation for multi-concept customization of diffusion models, Gu et al. 2024
[2] Orthogonal adaptation for modular customization of diffusion models, Po et al., 2024
Thank you again for your time and consideration.