Summary
This work introduces a novel framework for generating animated 3D objects from single-view videos. The proposed framework employs a feed-forward approach, thus eliminating the need for computationally expensive optimization. The core idea is to create a large-scale synthetic multi-view video dataset and train a corresponding 4D LGM model, which is initialized from the 3D LGM [49] and enhanced with temporal self-attentions. Additionally, an interpolation network is introduced to improve the frame rate further. Overall, this method achieves superior performance in producing 4D assets, both in terms of efficiency and reconstruction quality.
Strengths
- To the best of my knowledge, this is the first work to achieve feed-forward generation of 4D assets from a given video. The successful utilization of a large-scale dataset to tackle this challenging task may inspire further research in the community on related topics.
- The proposed framework demonstrates superior performance compared to existing methods across all evaluation metrics, as evidenced by the current evaluation (Figure 5, Tables 1). Additionally, the user study indicates a general preference for the results produced by this framework.
- Moreover, several significant ablation studies have been conducted on different components of the framework. Notably, the justification for using a pre-trained LGM (Figure 6(b)) is particularly convincing.
- Furthermore, the autoregressive generation method can produce animated objects with longer intervals, beyond the training time intervals.
Weaknesses
Technical contributions
- Despite its impressive performance, this work's technical contributions are somewhat limited. The two main contributions are temporal self-attention and the 4D interpolation network.
- The temporal self-attention can be seen as a direct extension of the multi-view self-attention proposed in [49].
- While the 4D interpolation network can speed up inference, its overall performance does not seem to have a substantial impact, as demonstrated in the supplementary material video.
Having said that, I agree that a straightforward solution to a new problem should be recognized, and the above weakness is relatively minor.
Questions
Some Questions:
- Regarding the temporal self-attention layer, why is only the time dimension considered in the self-attention mechanism rather than incorporating both multi-view and temporal information (full attention)? An analysis focusing on efficiency and performance would be valuable to understand the rationale behind this design choice.
- Why is the ablation study performed using PSNR (a reconstruction-based metric) instead of the metrics provided in Table 1?
- It appears that only qualitative comparisons are offered for the interpolation network. Could the performance be evaluated using the metrics shown in Table 1?
- Given that this work is an original contribution to the feed-forward generation of 4D assets, the community would greatly benefit from the release of the dataset and code to enable the reproduction of the work. Are there any plans to release these resources?
Limitations
- This work also necessitates a "synthetic" multi-view video dataset for training. According to the implementation details, generating this dataset takes approximately 3 days using 200 GPUs. While this is easier to obtain than a real dataset, it still demands significant computational resources. Including a more comprehensive discussion on this in the limitations section would be beneficial.
- The proposed framework is claimed to generalize extremely well to in-the-wild videos (Abstract). However, there are only a few samples provided (4 samples in Figures 1 & 4), and many of these are in an animated style, with the exception of the left example in Figure 4. This left example displays various artifacts on the produced objects, such as blurred arms. Given the current evidence, this claim appears to be inadequately supported, and it is recommended to either include more samples or lower the claim accordingly.