Summary
The paper introduces an object-centric representation for multi-object 3D pose control in image diffusion models. Instead of text token sequences, it utilizes Neural Assets, which are per-object representations learned by pooling visual features from reference images and reconstructing objects in target frames. This disentangles appearance and pose features by fine-tuning a pre-trained text-to-image diffusion model while conditioning on target frame poses, enabling fine-grained 3D pose and placement control in scenes. The results are demonstrated on both synthetic and real-world datasets in multi-object editing tasks.
Strengths
1. Object-centric representation: the proposed approach introduces a novel object representation to tackle conditional image generation, which disentangles 3D appearance and 3D pose. This representation provides a new solution to tackle similar tasks in controllable 2D image / 3D scene editing tasks.
2. Experimental Results: The experiments demonstrate the effectiveness of the proposed Neural Assets in achieving fine-grained control over individual object appearances and poses within a scene. Its application is shown in both synthetic and real-world scenes, supporting tasks like background swapping and object transfers across different scenes.
3. The paper is overall well written, providing good motivations and comparison with related work. Some concerns in the method section could be further clarified (see below).
Weaknesses
I value the performance the proposed model presented in manipulating objects while keeping the rest unchanged. I spent quite some time trying to comprehend how the current design leads to its performance, and still have the following questions, which might be helpful if they are discussed in the paper:
[-] The paper utilizes projection and depth of the absolute object poses in the pose embedding. This is kind of confusing as the it requires the model to encode how the object should look like in the canonical space. If this is the case, the model should be able to perform per-object reconstruction with the object-centric representation through images under different poses and even maybe object pose estimation. A common practice is to use the relative pose change, as the authors discussed in the background modeling.
[-] The authors leverage SD as base model, but one significant difference is that the proposed model put the learning bottleneck to the conditional module, compared to Zero-1-to-3 or ControlNet. All the useful features are embedded in the conditions now, including how the images should look like. In original SD, the conditional module is relatively light-weight. I'm wondering if there are limitations for the current design to capture the complex conditions and recover the input images.
[-] One related concern is that from the results, it seems the reconstruction results are also decent. I'm wondering what role is the randomness of the noise play in the current model. Will the model output different images if the seed changes?
[-] How should the model comprehend the objects in the target images that do not exist in the src images, especially during training?
Some other concerns:
[-] The model may lack a total understanding of the scene, especially the interactions between objects, e.g., when one object is supported by another or when the 3D bounding boxes of two objects collide with each other. How will the model react in such scenarios?
[-] The proposed model faces limitations in fully manipulating 3D object and their relationships as naturally as humans do. The requirement of 2D/3D bounding box pose challenges to the detection models and the manipulation as projection points is not intuitive as humans who commonly use language.
Questions
See above. Overall, I think the paper offers a new perspective for object-centric learning, especially under the image generation settings. I still have some concerns and will consider raising my rating if the rebuttal resolves them.