Summary
This work proposes an approach for 3D generation from a single input image. Given an input image, a monocular depth estimate and an estimated instance mask, an image constrained radiance field is optimized following the score distillation approach from DreamFusion. They key idea is to enforce a hard constraint on the radiance field that guarantees that the input image can be reconstructed exactly whereas existing methods only encourage this with a reconstruction loss.
Strengths
The paper is easy to follow and well-written. Both, qualitative and quantitative analysis indicate that the proposed method outperforms the existing baselines.
In general, inferring 3D representations from single input images is an interesting and challenging task with high relevance to the community. The idea of integrating the input image as a hard constraint in optimization is intuitive and might be interesting to the community.
Weaknesses
My main concern is that the central claim of the paper, that integrating the image as a hard constraint over using a reconstruction loss makes the training more robust and that “The proposed ConRad representation significantly simplifies the process of a generating a consistent 3D model for the input image” (L.55), requires more experimental support. While the supplementary video shows a few failure cases for the baselines these results could be hand-picked. Importantly, it is also not clear if the stable training indeed results from using the proposed image conditioned radiance fields. For this, I would have liked to see an ablation study where the training pipeline is exactly the same except that once an image conditioned radiance field is used (hard constraint) and once a reconstruction loss is used (soft constraint), also ablating different strengths/weights of the reconstruction loss.
Using pearson correlation as a depth loss for monocular depth was already proposed in [1] which is not cited. Further, [1] is missing from the baseline comparison and while there is no code available, a qualitative comparison could be performed similarly to the other baselines, i.e. using results from the original paper.
[1] Deng et al, “NeRDi: Single-View NeRF Synthesis with Language-Guided Diffusion as General Image Priors”, CVPR2023
Questions
Please also see Weaknesses.
L.262 mentions that NeuralLift360 uses mask supervision. However, this was not clear to me from the original NeuralLift360 paper. Could you please explain again how NeuralLift360 uses foreground masks and provide a pointer where this is stated in the original paper?
Your approach seems to achieve a higher image fidelity than the baselines. Is this only a result of more stable training or which component, e.g. compared to NeuralLift360, enables the higher image fidelity?
Minor:
L.277 “improved metrics”, it is not shown that the metrics improve over existing metrics, so I suggest removing “improved”
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Limitations
In general, the limitations and the broader impact were adequately discussed. From the supplementary video it looks like sometimes objects are not fully opaque (bird). This should be added to the limitations section.