Summary
This paper introduces a method to improve cross-view consistency in multi-object NVS using pre-trained diffusion models. MOVIS enhances model inputs with depth and object mask information, adds an auxiliary task for predicting object masks in new views, and uses a structure-guided timestep sampling scheduler to improve object placement and detail recovery.
Strengths
1. Targeting multi-object NVS is interesting, as most current NVS methods are validated on single-object scenarios. By addressing challenges like correct object placement, shape, and appearance across views, this direction opens up new possibilities for compositional scene generation.
2. Using corresponding points as a validation metric is a valuable addition, as it provides a more robust measure of cross-view consistency. Unlike image-level metrics alone, corresponding points offer a way to directly assess if objects maintain accurate positions, shapes, and appearances across different viewpoints. This metric helps validate the model’s effectiveness in capturing structural relationships and ensures synthesized images maintain coherent spatial relationships across views.
Weaknesses
1. The paper lacks commonly adopted NVS evaluation metrics that directly assess multi-view consistency, such as running 3D reconstruction (like nerf) to compute mesh differences (e.g., Chamfer distance) or re-rendering metrics like SSIM or LPIPS. These approaches provide a more direct evaluation of the generated multi-view consistency by quantifying structural and perceptual alignment across views. Without these metrics, it’s harder to objectively compare the quality of synthesized outputs.
2. While the paper references several recent improvements on Zero123, including methods like SyncDreamer and Consistent123, it doesn’t experimentally compare MOVIS with these state-of-the-art techniques. Given that these methods also tackle cross-view consistency NVS challenges, a direct comparison would provide valuable insights into how MOVIS performs relative to these advances, strengthening the claims about its effectiveness.
3. The paper could improve fairness in comparisons by controlling variables like training data and strategy, which currently differ from the baseline methods. Using consistent training data and experimental protocols would ensure a fair comparison. A suggested approach includes: 1) selecting a subset evaluation dataset from the Objaverse validation set with multiple objects; 2) training MOVIS on this dataset using its proposed strategy; and 3) evaluating MOVIS and baseline methods on this subset. This setup would yield more robust and convincing results by isolating the effects of MOVIS’s architectural and training improvements.
Questions
1. In Table 1, finetuning Zero123 on the 3D-FUTURE dataset seems to unexpectedly worsen its performance, which is unusual as fine-tuning typically enhances or maintains model quality. Could the authors elaborate more on their fine-tuning process on the new dataset? This result may indicate potential overfitting, which is often avoidable with techniques like regularization, LoRA, data augmentation, or early stopping. For a fairer comparison, consider integrating 3D-FUTURE into the original training data (Objaverse-10M and Objaverse-800K) and training Zero123 jointly. This approach would help assess the true benefit of adding 3D-FUTURE and prevent overfitting by diversifying training instances across datasets.
2. For the Zero123 comparisons, is the Zero123-XL version being used? This clarification is important, as the standard and XL versions have different capabilities and model sizes, which could influence performance. Comparing MOVIS to Zero123-XL, if it’s not already the case, would provide a stronger benchmark and ensure that results accurately reflect improvements attributable to MOVIS rather than differences in model scale.