Slot-guided Volumetric Object Radiance Fields

Summary

This paper presents slot-guided volumetric object radiance fields (sVORFa), a learning method for 3D object-centric representation. They proposed three key techniques, scene decomposition based on global pooling and slot-based representations, hyper-network to generate per-object radiance fields, and scene composition module. They have tested the method on various tasks on synthetic datasets, such as CLEVR and Room, showing superior performance to prior arts. Finally, they provided the simple experiment results on real images (LLFF), highlighting the broader effectiveness of the methods.

Strengths

1. The proposed method improved the prior methods by a large margin, especially on Room-Chair and Room-Diverse datasets (segmentation task) 2. The newly suggested modules seem very efficient. and I think the global pooling features combined with the slot-guided method is a reasonable and better approach than the previous slot-attention method, at least in these particular tasks.

Weaknesses

1. From my understanding, three modules were newly adopted or introduced to object-centric representation learning tasks. They are 1. ‘global pooling’ with slot-guided scene decomposition module, 2. hypernetwork, and 3. scene composition modules. However, throughout the experiments, I do not see which components are actually effective compared to prior methods. For example, they could have introduced ‘hypernetwork’ to uORF, or they could have tested ‘slot-attention’ in the proposed sVORF to see the effectiveness of the 'hypernetwork' or ‘global-pooling’ slot-guide method. Hence, it is hard for the readers to know every component actually contributes to the final performance. 2. It seems connectivity regularization (CR) is very important, I would like to see how much CR can improve the performance of the previous methods, such as uORF and others. 3. Similarly, I believe we can easily incorporate spatial broadcast (SB) or slot mixers (SM) in the proposed method sVORF, which could clearly show the effectiveness of the proposed scene composition module. 4. Although the newly introduced modules are well orchestrated into a single model and improve performance, I do not see significant technical novelty.

Questions

1. In Table 2, why did you have to use different settings and re-evaluate the previous methods? Other questions are embedded in the weaknesses section.

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

I do not see any potential negative societal impact from this work.

Summary

This paper proposes a method, sVORF, for unsupervised object-centric 3D representation learning. Given a single image as input, it decomposes the scenes into individual 3D objects and 3D backgrounds which thus support object segmentation, novel view synthesis (named scene generation in the paper), and scene editing (including object moving and background repainting). The key idea of the paper is to combine object slot with object radiance fields together in a framework, in which the object slot is responsible for decomposing the scene into objects. In contrast, the object radiance fields are used to re-compose the scene for volumetric rendering. The only cue for training the network is the rendering loss between the re-composed and volumetric rendered image with the input single-view image. The overall performance of sVORF is plausible and thorough experiments and ablation studies well validated the effectiveness of the proposed method.

Strengths

+The task of reconstructing each object and the background in 3D given only a single RGB image is very challenging and with great importance for practical applications in robotics and CHI, etc. +The overall performance of the proposed method is strong when compared with existing approaches. +The idea of using the attention mechanism for scene decomposition and re-composition as well as the use of HyberNetwork for object-level NeRF generation is sound and practical. +The demonstration of segmentation on real captured dataset LLFF is important to help the readers to qualify the performance of the method. +The paper is well written and easy to follow.

Weaknesses

- The dataset used for validating the proposed method is quite simple, with obvious color or shape differences between objects and simple backgrounds, which is very convenient for segmentation and decomposition. In the LLFF Dataset, the segmented foreground color is also very different from the background. So maybe more complex scenes should be used to demonstrate the effectiveness of the proposed method. - The HyperNetwork produces MLP parameters of NeRF. However, the MLP representation of NeRF is very ambiguous, so learning such a hyberNetwork for NeRFs may not be very robust, please explain how to deal with this problem, especially for real-world scenes with complex objects in it. - In Fig.5, only segmentation results are provided on the LLFF dataset, is it possible to also provide the NVS results? why not? - The main paper should clarify the training strategies and key implementation details like how to set K for each dataset.

Questions

- Line 224, please provide the full name of FFN since this is the first time you use FFN. - In Table 2(a), it seems ObSURF works generally better on 3D segmentation, does this mainly benefit from the depth input? - Line 149, a->an

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

Yes.

Summary

This paper presents a method called sVORF for learning 3D object centric representations. A transformer decompose a single input image into slots which are then each mapped to volumentric radiance fields by a hypernetwork. These object-radiance fields are then composed into a 3D scene. The effectiveness of sVORF is demonstrated on a set of simple synthetic datasets and two real world scenes. It achieves SOTA or at least competitive results on CLEVR567, CLEVR3D, Room-Chair, and Room-diverse datasets, compared to SlotAttention, uORF and COLF.

Strengths

* The problem of unsupervised 3d object decomposition is quite challenging. The fact that sVORF performs so well with only a single input image, and no depth or segmentation supervision is remarkable. * The writing is clear and easy to follow. The figures look nice, are helpful and are easy to understand. * The individual parts of the method are well motivated and their individual contributions are established through systematic ablations.

Weaknesses

* The paper stresses the low computational costs and memory footprint of sVORF, but fails to report any data on them in the main paper. Memory consumption is mentioned in the appendix, alongside training time per epoch. But information about the length of training is lacking so the real computational costs remain unclear. * The studied scenes are visually simple (yes even Room-diverse which is advertised as complex). The only exception are the two scenes from LLFF. It is encouraging that sVORF succeeds on these two scenes, but in this setting it appears to me that the model is effectively overfitting on the given scene, and there is no evidence for generalization to novel scenes in that setting. * The restriction to a single input image is rather limiting. The 3D structure of a general scene is highly underspecified from a single image, which should lead to large uncertainties (e.g. in occluded areas). For simple scenes like CLEVR and Room-Chair this is not a problem because none of the elements is ever fully occluded, and they come from a small set of known objects. But I expect this to become a severe limitation for scaling to more complex scenes. Especially since sVORF and its training objective are not designed to be probabilistic / generative, and its fidelity will thus likely suffer substantially from complex ambiguities.

Questions

- How many samples are used per scene during training. Appendix A mentions 64 rays with 64 coordinates in the coarse volume and 128 additional coordinates in the fine volume. What are the coarse and the fine volumes? - For how long (number of updates / epochs) was sVORF trained? How does that compare to uORF, COLF, OSRT? - The LLFF results use only two scenes. Does that mean the shown results are training images? Is sVORF trained on both scenes at once or as two separate models. In the latter setting it seems to me that sVORF is essentially equivalent to a simple volumetric NeRF, since the model could in principle learn to produce the correct target views during inference without even looking at the input view. Is there any evidence for generalization in this setting?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

3 good

Contribution

3 good

Limitations

There is a brief discussion of limitations in the supplementary that touches on two important limitations of the method. It does however not discuss any concerns regarding generalization to more complex real-world scenes.

Summary

This paper introduces sVORF, a compositional method for representing scenes as a collection of slots that are parametrized as per-object radiance fields. The proposed model can be used to perform novel-view synthesis, as well as semantic segmentation in 3D. Moreover, the proposed compositional representation, also enables performing simple editing operations in the scene. In more detail, given an input image, sVORF first extracts image features from the image and passes them to a transformer encoder-decoder module that extracts slots for the objects in the scene and the background. Subsequently, these slots are mapped to volumetric object radiance fields using a hypernetwork. Using the per-object radiance fields, the scene is rendered by simply compositing their outputs at a 3D location with the guidance of each slot. Unlike [8] that was the first work that explored discovering slots in 3D data, sVORF maps slots to per-object radiance fields using a hypernetwork. The authors evaluate the performance of their model on the CLEVR-567, CLEVR-3D, Room-Chair, Room-Diverse and the LLFF datasets and compare with several compositional representations that represent slots as radiance fields [8, 35] or as light fields [9]. From their experimental results, it seems that the proposed model outperforms all baselines both on the novel view synthesis task as well as on the 3D segmentation task. Overall, I think the proposed idea for discovering slots in 3D is interesting and the authors show that their approach is working for simple scenes with a few objects. In terms of novelty, the proposed pipeline is quite similar to [8], with the main difference being the use of the hypernetwork for mapping slots to per-object radiance fields. Looking at the quantitative results, it seems that the proposed model is better than the baselines on most tasks, thus I am leaning towards accepting the paper but I am still a bit concerned by the fact that the proposed model and [8] are relatively similar and that the proposed model is only evaluated on extremely simple scenes.

Strengths

1. The authors conduct multiple experiments and compare their model with several baselines on multiple datasets. From the quantitative evaluation, we note that the proposed model outperforms all baselines on most tasks. I particularly liked the scene editing experiment in Sec. 4.3. 2. Although some things are not 100% clear I think the paper is easy to read and relatively nicely written. I really liked Figure 1 that is a pictorial representation of the overall pipeline.

Weaknesses

1. One of the major weaknesses of this work is that the authors only demonstrate the performance of their model on very simple datasets that consist of few objects that in most cases belong to the same object class e.g. chairs. Although I am aware that all baselines evaluate their models on similar setups I think it would have been great if the authors could show that their model works on more challenging scenarios, with different objects and more diverse backgrounds. A rather simple indoor dataset that the authors could consider is the 3D-FRONT dataset. Another less exciting alternative would be to also show results on the City-Block dataset introduced in [9]. However, since also this dataset contains scenes with cars, I think that the 3D-FRONT would be a more appropriate benchmark. 2. In Sec. 3.2 and in L116-117 the authors state that the adopt an efficient transformer module to infer the object and the background slots form the image features. Can the authors describe how this efficient transformer module compares to the classical slot attention module used in [8]? Is the efficient transformer module a standard transformer encoder/decoder? I think that to make things very clear, it would have been very useful to properly define (e.g. using a math expression) how is the cross-attention defined? In addition, can the authors also clarify why/how is this model efficient? 3. Although, the authors provide several ablation studies, I believe that they should have ablated the impact of using a hypernetwork as opposed to directly using the slots for conditioning the per-object radiance fields. This is more similar to [8] and it could better explain why is the proposed model outperforming [8], although their quite similar. 4. I am bit concerned regarding the limited technical contribution of this work. Unless I am missing out something I think the proposed paper is quite similar to [8]. That being said, the proposed method seems to be significantly better than [8]. This could be either due to the use of the hypernetwork or the use of the attention module for inferring the slots. I believe that it is very important to clarify what are the differences and also try to justify the improvement in performance though ablations e.g. ablate the use of a hypernetwork for mapping slots to per-object radiance fields.

Questions

1. Unless I am missing out something, I think that the proposed model assumes that the number of objects/slots is known right? In other words, although we might not know what are the objects, we know how many objects/slots exist in the scene, right? This is never clearly mentioned in the text. I think the authors should clarify it. 2. I think that the per-object NeRFs should be locally defined, namely each NeRF is augmented with a local affine transformation defining its pose in 3D. Otherwise, it would not be possible to change the location of a specific object/slot to a new position. Is this really the case? 3. I think that from the provided information for the composing mechanism in L146-150 it is not 100% clear how are the aggregation $D^c$ and the attention bloc $D^a$ really implemented. One thing that is not very clear to me is why do we need both? Can the authors please explain? 4. In L264-272, the authors present an ablation study that tries to investigate the impact of the Novel View Synthesis setup. To be honest, I am not 100% sure that I understand what the authors mean by "we modify our reconstructed target view to equal with the input view". Can the authors please clarify? 5. I am wondering what is the quality of the 3D geometry of the proposed model and how does it compare to baselines. Since the authors represent the scene using a compositional NeRF-based representation I think it would be useful to also show some depth maps at least in the supplementary.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

The authors discuss the limitations of their work in their supplementary material but I was not able to find any discussion about the potential negative societal impact of their work. That being said, I think that for this paper this discussion is not 100% necessary.

Summary

This paper proposes sVORF to tackle the challenging problem of ‘unsupervised’ scene decomposition by learning a series of slot-guided neural radiance fields. Inferring object-level NeRF from a hyper-network instead of costly GRU modules, the overall pipeline can be efficiently trained with on a collection of multi-view images and produce promising segmentation results on several synthetic benchmark datasets.

Strengths

- The overall ideas is well motivated and easy to follow, and the main pipeline could be well taken. - Extensive experiments on adequate synthetic datasets are conducted. The numerical results are promising compared to SOTA baselines, with detailed ablations.

Weaknesses

- Limited unique contribution compared to previous work like uORF. I think this paper proposes a good extension on top of uORF while my main concern is that the unique contributions are somewhat limited. The overall pipeline is built on top of previous work uORF by replacing the expensive GRU modules to hyper-network inferred compositional object-level NeRFs, which I see as the main difference bringing efficiency advantages. - I do appreciate the promising performance on synthetic datasets, however I think more discussions and results towards the main statements (consistency, efficiency against current baselines) need to be further strengthened. - The method of using radiance fields instead of light-fields stand out in terms of strict multi-view consistency. Therefore, discussions and analysis of multi-view consistency of segmentation masks (on synthetic and real-world cases) are expected to show to validate this point. - Are the adopted backbone networks (ResNet34 and ViT-Base of real-world image segmentation) pre-trained in any way or trained from scratch (L318)? As I am not sure if the used datasets are sufficient to train such large network. If so, is the method strictly unsupervised anymore? More clarifications are expected. Also, for real-world segmentation, I am wondering how does the backbone impact the performance if ResNet34 is used. - How does appearance affect the perform? Does the mode really learn to decompose the scene intrinsically or a very good ‘colour/apperance’ segmentor? I see some good evidence from fig2 but find the networks struggles to separate close-by objects with similar colour as well as shadows. Moreover, related to last point of consistency, is this ‘incorrect pattern’ also consistent across view? - Related to last point, visualization of learned object-level radiance fields are expected. Readers could know if the method indeed decompose the scenes into objects and background as we expect. - It would be more exciting to see how the methods work on real-world images with more than 1 foreground objects. - What is the total training time required to converge sVORF as only per-epoch time is provided. -L118. (GEU) --> (GRU)?

Questions

Please see the weaknesses above.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Limitations have been discussed in the supplement.

Response to author rebuttal

Firstly I thank authors for prodiving more clarifications and experiments on relatively more complext MSN dataset. I have carefully read all the reviews and author response, overall I think some concerns have been succussfully addressed. I think the additional visualisation and analysis would be critical for people to see how the proposed methods decompose the scenes via slots related neural fields. In addition, I expect the detailed configuration as well as experiments on gray-scale images and MSN dataset to be fitted in the main paper and believe it will be a stronger submission. However, some key limitations still remian, for example, I guess the proposed method as well as previous ones would struggle on real-world scenes with multiple objects with rich textures. I hope such limitations could be clearly discussed in the main paper to make it clear where the method stand in the challenging task of unsupervised scene decomposition. Overall, I raise my score to 5 with borderline acceptance considering above modifications and limitations.

Response to rebuttal

First, I would like to thank the authors for writing such an extensive rebuttal with numerous additional experiments. I agree with the other reviewers, in that the paper is more of a recombination and improvement upon previous works than an innovation in and of itself. I am also still skeptical about the ability of sVORF to generalize to data with real-world complexity. However, I would like to point out that the problem of unsupervised scene decomposition is extremely challenging, and that the authors do demonstrate clear improvements above the baselines, especially in terms of compute. It is still not a strong result, but in my opinion the additional results and clarifications significantly improve the paper and push it into the region of a clear accept.

Rebuttal Acknowledgment

I would like to thank the reviewers for taking the time to provide additional results on more complex scenes as well as discuss the key differences between the proposed model and [8]. After having read the author's rebuttal and the other reviews, most of my concerns have been addressed, hence I will raise my score to 6: Weak Accept. That being said, I would like to urge the authors to include the additional experiments, provided in the rebuttal, in the final version of their paper. If possible, please also consider providing additional qualitative comparisons on the various datasets. Moreover, I think that the authors should also add a section discussing the differences of their model compared to [8]. I believe that the ablation study about the impact of the use of hypernetwork is very important and the authors should definitely include it in the final version of their paper.