Human visual reasoning is characterized by an ability to identify abstract patterns from only a small number of examples, and to systematically generalize those patterns to novel inputs. This capacity depends in large part on our ability to represent complex visual inputs in terms of both objects and relations. Recent work in computer vision has introduced models with the capacity to extract object-centric representations, leading to the ability to process multi-object visual inputs, but falling short of the systematic generalization displayed by human reasoning. Other recent models have employed inductive biases for relational abstraction to achieve systematic generalization of learned abstract rules, but have generally assumed the presence of object-focused inputs. Here, we combine these two approaches, introducing Object-Centric Relational Abstraction (OCRA), a model that extracts explicit representations of both objects and abstract relations, and achieves strong systematic generalization in tasks (including a novel dataset, CLEVR-ART, with greater visual complexity) involving complex visual displays.
Paper
Similar papers
Peer review
Summary
The paper proposes to combine an object-centric representation model (Slot Attention) with a relational reasoning module to create the Object-Centric Relational Abstraction (OCRA) model. For this model, Slot Attention is first pre-trained in an unsupervised way to represent objects separately. Then, the relational reasoning module, consisting of a "relational bottleneck" and a transformer model, is applied to these object-centric representations. This reasoning model is trained in a supervised way on a subset of possible objects. The experiments on three different datasets show that this approach generalizes better to previously unseen objects than comparable baselines.
Strengths
The paper is written well and easy to follow. The introduction provides a strong motivation for this line of research. The proposed approach and conducted experiments are described clearly and with sufficient detail, such that it should be possible to reproduce them. The proposed approach seems to be novel and achieves strong generalization performance on previously unseen objects.
Weaknesses
In my eyes, the main weakness of this paper is that its contribution relative to existing work is not entirely clear. 1) Most importantly, the paper positions itself as proposing an approach with a stronger inductive bias toward relational abstraction, which ultimately leads to better generalization performance (for example, introduction (l.46) and related work section (l. 135, l. 170). However, the experiments do not compare against many existing approaches. Without such a comparison, it remains unclear whether the existing approaches actually suffer from the problem that the proposed approach claims to solve. Thus, ideally, the paper should provide such a comparison, or alternatively adjust the positioning of the proposed approach relative to existing work. 2) Besides this, it remains somewhat unclear which parts of the proposed approach are building on existing work (such as Slot Attention) and which parts are new. For example, do equations (1) and (2) and sections 2.2 and 2.3 describe novel architectural components or have they been used in a similar fashion before? --- 3) The related work section fails to cite many relevant papers. Object-centric representation learning has seen growing interest in recent years, with many papers proposing various solutions. I would recommend taking a look at the related works section of [1], which provides a comprehensive overview of papers that are relevant to this work. Besides this, [2] describes a recent model that combines object-centric representation learning with relational reasoning capabilities and [3] describes generalization properties of Slot Attention to previously unseen objects. 4) The experimental results could be strengthened by applying all baselines across all datasets. Additionally, why is the CoRelNet not included in the baselines? It seems like one of the most relevant existing approaches. --- Minor points: 5) Equation (2): What does adding $pos$ to the position embeddings $m_k$ add for the relational reasoning module? It is not dependent on the input, and cannot be adjusted by the relational reasoning module as it is part of the pre-trained, frozen SlotAttention model. Thus, I would expect that it doesn't provide much useful information. 6) Equation (4): $m_k$ will implicitly contain some information on the object shape, since $attn$ is used in Eq (2). Do you think this could allow the model to circumvent the relational bottleneck, if $m$ would be processed by some more powerful non-linearities? 7) Line 115: "endowing OCRA with an explicity variable-binding mechanism". What do you mean with this statement? 8) line 67: what is a "position-wise fully-connected layer"? Are you referring to the 1x1 convolutions in Table S2? In general, I think it would be helpful to draw direct connections between the components of the architecture described in Table S2 and the components described in the main text. 9) Eq. (1) + (2): what is the $\cdot$ symbol referring to here? In Eq. (3), this symbol is used to describe a dot procuct, which would not make much sense here, I think. 10) Lines 255-262: I would move most of this description into the method section. 11) Table 2: Use the same order for the different tested conditions in the table as is used within the text. --- [1] Jiang, J., Deng, F., Singh, G., & Ahn, S. (2023). Object-centric slot diffusion. arXiv preprint arXiv:2303.10834. [2] Wu, Z., Dvornik, N., Greff, K., Kipf, T., & Garg, A. (2022). Slotformer: Unsupervised visual dynamics simulation with object-centric models. ICLR 2023 [3] Dittadi, A., Papa, S., De Vita, M., Schölkopf, B., Winther, O., & Locatello, F. (2021). Generalization and robustness implications in object-centric learning. ICML 2022
Questions
As described above, I believe the main weakness of this paper is its current positioning relative to the existing research landscape. If the authors could improve this point, or strengthen the current claims by providing additional experiments, I would be happy to adjust my score.
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
3 good
Presentation
3 good
Contribution
2 fair
Limitations
While the relational bottleneck improves the model's ability to generalize to previously unseen objects, I would expect it to also limit the model's applicability to other settings. For example, in the experiments, a separate model is applied for each separate task and thus for each separate relation to be learned. I would expect that the relational bottleneck would harm the model's performance if it had to differentiate several relations at once. Additionally, I would expect it to be more difficult to represent more complex relations between objects, as the slightly worse results on the spatial relations in the SVRT dataset might indicate.
Summary
- The paper seeks to tackle various visual reasoning problems with a focus on systematic generalization. - The paper argues that we need explicit inductive bias to extract object-object relationships and express these relationships via a low-capacity representation. - The paper seeks to do this by first extracting object vectors (or slots) using pre-trained slot attention. It then performs a dot-product of these slots to obtain a “relation embedding” — which is really a scalar number expressed as an embedding (as far as I understand). These relation embeddings are then given to a transformer and the whole model (except the slot attention) is trained to classify whether the given input is conformant to the true pattern or not. - The performance is then evaluated on a systematically OOD test set and compared with various baselines and model ablations on 3 visual reasoning benchmarks: ART, SVRT, and CLEVR-ART. Here, CLEVR-ART is a novel dataset proposed within this paper.
Strengths
1. Simple and elegant architecture. 2. Propose a new dataset for visual reasoning and systematic generalization called CLEVR-ART, a sort of visually complex variant of ART. This appears to be a useful contribution to facilitating progress in the community. 3. Good systematic generalization performance compared to baselines. 4. Several useful baseline comparisons. For instance, the paper shows that a standard transformer when applied to slots is not enough to systematically generalize and the proposed relation layer is useful. 5. Useful ablations. For instance, the paper shows that all of the following are useful: relation bottleneck, relation embedding, object-centric inputs, decomposing position and appearance, etc.
Weaknesses
I have several questions and possible avenues for improvements that I describe in the “Questions” section.
Questions
1. Also, what part of the architecture is enforcing the “relational bottleneck”? Is it due to the fact that the dot product of two slots results in a scalar number? If yes, then this should be highlighted somehow both in the text and perhaps also in Figure 1. One may also conduct an experiment in which the “bottleneck” dimension is gradually increased (e.g., 1 to 2 to 5 and so on) and show that this gradually worsens the generalization performance. 2. Line 30: Regarding learning relational abstractions: Is it possible to take a large batch of inputs, extract relation “embedding” and visualize object pairs that have similar relational “embedding”? This would give more credence to the fact that relational abstractions are indeed being inferred. 3. Line 108: Is it necessary to say “shared” here? IMO, I wouldn’t have assumed that the projection matrices were not shared (based on the equations). 4. It should be better highlighted in the introduction section and the abstract that CLEVR-ART is a novel contribution. This should also be discussed in the related work relative to the existing visual reasoning benchmarks. It would also be useful to make a statement about whether this dataset will be released to the research community or not. 5. Table 1: Why are several baselines shown for ART not shown for CLEVR-ART and SVRT? I think these baselines will be useful to show for all datasets. Also, I would suggest using a consistent format for reporting the results of all 3 datasets i.e., picking either the bar-plot format or the tabular format. 6. L263: In the case of inputs involving multiple images, how are they processed by the network? As I understand, the model does not sequentially consume multiple images. Does “inserted” mean programmatically generating the candidate images, each containing multiple objects? 7. L283-298. I find the discussion of the results rather small. For instance, what is the rationale for the comparison with ESBN or GAMR? What is the key distinguishing characteristic of those baselines with respect to the proposed one? What can we learn from the result of this comparison i.e., why does the proposed model outperform? 8. (Minor) Line 34: Would be good to cite the paper(s) that support this statement “By biasing architectures to process visual inputs in terms of relations between objects, these recent approaches have achieved strong systematic (i.e., out-of-distribution) generalization of learned abstract rules, given only a small number of training examples.” I am giving a score of 6 in the hope that the authors will try and address some of my questions and concerns. If addressed to some degree, I will happily increase the score.
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
4 excellent
Presentation
4 excellent
Contribution
4 excellent
Limitations
Yes, the limitations are discussed in the last line of the conclusion.
Summary
The research topic of this study is the development of a learning machine that achieves systematic generalization in reasoning over complex relations of objects in a visual input (still image). Toward this goal, the authors proposed a new neural network model (OCRA) taking inspirations from the recent results on effective inductive biases for systematic generalization in relational reasoning and methods for obtaining object-centric representations. More concretely, the proposed model comprised of three core components: first component (slot attention mechanism) to extract object-centric representation from a visual input containing multiple objects, the second component to compute pairwise relation embeddings, and the third component (transformer) to provide the final output related to the higher-order relations. The effectiveness of the proposed model was tested with three visual reasoning tasks, two existing and one new, and also compared with various baseline models. As a whole, the proposed model can be said the best in terms of systematic generalization among the compared models. An ablation study was also conducted to evaluate the roles of the components in the proposed model and pretraining.
Strengths
The top-level idea behind the proposed model, that is, combining the inductive biases for relational reasoning and a method for obtaining object-centric representations is clear and reasonable. How to implement the idea with the three core components are explained fairly well. Although the top-level idea might look somewhat straightforward at first glance given the advancement in the two directions, the concrete implementation is not trivial, and the differences from the existing work are described in the paper. The effectiveness of the proposed model was tested with fairly rich experiments. Two existing task (ART and SVRT, both created with 2D shapes) and a new task developed based on the CLEVR (CLEVR-ART, created with 3D shapes) were used and various baseline models including a very recent one (GAMR, accepted at this year's ICLR) are compared with the proposed model. An ablation study gives additional value to this work. Development of a learning machine that achieves systematic generalization in complex visual reasoning is an important topic in AI. Although there is still a distance to the real-world applications as stated in Section 6 (Conclusion and Future Directions), the proposed method and the evaluation results will be of interest to the NeurIPS audience.
Weaknesses
1. A weak point of this submission is lack of the source code of the proposed model as Supplementary Material. It is also unkown whether the source code will be made publicly available if the paper get accepted. These points cast a shadow on the reproducibility. 1. A relatively weak point of the proposed model itself seems to be around the number of slots, $K$. First, as stated in line 117, the proposed model computes all $K^2$ pairwise relations. Second, in the current model, $K$ should be defined in advance. These characteristics can be obstacles in the reasoning over the relations of objects in the real-world visual inputs. Namely, the fist point might affect the applicability (scalability) to complex natural images and there should be some additional mechanism to determine an appropriate $K$ in the first place if the current structure of the proposed model is kept. 1. (Related to 2.) Although it is stated that there exists gap between the problems addressed in this study and the real-world vision in Section 6 (Conclusion and Future Directions), the details are not explained.
Questions
Questions 1. Can the source code of the proposed method be provided at the Author Rebuttal period? In addition, will the source code made publicly available if the paper get accepted? 1. What are your thoughts on the second point in Weaknesses section above? Suggestion It would be beneficial if the authors could add detailed descriptions about what kind of differences between the problems treated in this study and the real-world vision should be overcome, and also add explanations about the relation between those differences and self-supervised learning methods mentioned in Section 6 if possible.
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
3 good
Presentation
3 good
Contribution
2 fair
Limitations
The authors stated in Section 6 (Conclusion and Future Directions) that the problems addressed in this study are still simple compared with the the real-world vision. However, currently it is not explained in details what kind of differences between the problems treated in this study and the real-world vision should be overcome. Please also refer to the Weaknesses and Questions sections above.
Summary
This work proposed a new method, named OCRA, that combines object-centric presentation learning (for object abstraction) and a relational bottleneck (for relational abstraction). Particularly, OCRA consists of three components: 1) a slot attention to extract object level representations, 2) a relational operator to get pairwise visual relations, and 3) a transformer to model higher-order relations. The slot attention model is pretrained on a large dataset and the relational modules are trained on a small task-specific dataset while freezing the slot attention. Experiments were performed on three datasets: ART, SVRT and CLEVR-ART to show the effectiveness of the proposed method.
Strengths
1. The idea of combining slot attention with a relational bottleneck (relation operator and transformer) for solving visual relational reasoning problems sounds interesting and also novel to me. 2. The presentation of the idea and the overall writing are very clear. 3. Experiments on synthetic datasets (ART and SVRT and CLEVR-ART) shows the proposed method can work for various relational reasoning tasks and it achieves better performance than baselines in many cases.
Weaknesses
1. For the benchmarks, I have a concern about their simplicity. For example, the performance of many methods on the ART dataset is close to 100%. Similar observations also exist in other two datasets. Does it mean we already have achieved human-level visual reasoning performance or we need better benchmarks to evaluate the success of methods? I think considering some more challenging benchmarks will make the results more convincing. For example, RAVEN [1] and Bongard-HOI [2] are two challenging benchmarks for testing a model's visual relational reasoning abilities. In particular, Bongard-HOI considers the real-world natural images. 2. In Figure 4, it is a little surprising that OCRA performs worse than ResNet on SVRT - spatial relations with 1000 training examples. Any intuition on this? 3. For the ablation studies, to test the impact of slot attention, it is too naive to do “feature map divided into a 4x4 grid”. How about comparing slot attention with some standard SSL trained object-centric representation learning methods? Also, from Table 2, we can see that Transformer has a much higher impact than the slot attention and relational bottleneck on both RMTS and ID. Does it mean the most important part of OCRA is the transformer rather than slot attention and relational bottleneck? If so, it downplays the method's significance a little, in my opinion. [1] RAVEN: A Dataset for Relational and Analogical Visual rEasoNing, CVPR 2019. [2] Bongard-HOI: Benchmarking Few-Shot Visual Reasoning for Human-Object Interactions, CVPR 2022.
Questions
My major concerns and questions are in how the experiments support the effectiveness and significance of the proposed method. Please see the weaknesses part for more details.
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
2 fair
Presentation
3 good
Contribution
2 fair
Limitations
The authors have well addressed the limitations.
Thank you for your response. I propose that the authors incorporate their responses to points 1 and 2 into their paper. I believe that doing so will better highlight its contributions. Moreover, the limitations I previously noted could enrich the paper if acknowledged explicitly. All my other concerns have been addressed adequately, and I will update my rating to "weak accept".
Further revisions
We thank the reviewer for these suggestions. We agree that adding additional discussion of these issues would further improve the paper. To address this, we have made the following revisions: - At the end of Related Work, we have added the following statement clarifying the connection between the baselines and previous work (Note that the references are those that were cited on lines 135 and 170 of the original submission): ‘To empirically evaluate the importance of the relational bottleneck, we compare our approach with a set of baseline models that capture the key computational properties of previous models, including a slot-transformer baseline that combines slot attention with a transformer reasoning module [9, 34, 46, 25], and a slot-interaction-network baseline that combines slot attention with an interaction-network reasoning module [43, 39, 15, 47, 2, 19, 35, 41, 12, 36].’ - We have also added the following statement to the end of Related Work, to clarify the novel aspects of our proposed approach: ‘To summarize our contributions relative to previous work, our proposed model includes: 1. A novel object representation format that is factorized into distinct feature and position embeddings (Equations 1 and 2), enabling a form of explicit variable-binding. 2. A novel relational embedding method that implements a relational bottleneck (Equations 3 and 4). 3. An architecture that combines these elements with preexisting components (slot attention and transformers) in a novel manner to support systematic visual reasoning from complex multi-object visual displays.’ - Finally, we have now revised the Conclusion to more explicitly address the limitations identified in this and other reviews: ‘6. Limitations and Future Directions In the present work, we have presented a model that integrates object-centric visual processing mechanisms (providing the ability to operate over complex multi-object visual inputs) with a relational bottleneck (providing a strong inductive bias for learning relational abstractions that enable human-like systematic generalization of learned abstract rules). Though this is a promising step forward, and a clear advance relative to previous models of abstract visual reasoning, it is likely that scaling the present approach to real-world settings will pose additional challenges. First, real-world images are especially challenging for object-centric methods due to a relative lack of consistent segmentation cues. However, there has recently been significant progress in this direction [33], in particular by utilizing representations from large-scale self-supervised methods [50, 7], and it should be possible to integrate these advances with our proposed framework. Second, the current approach assumes a fixed number of slot representations, which may not be ideal for modeling real-world scenes with a highly variable number of objects [51]. Though we did not find that this was an issue in the present work, in future work it would be desirable to develop a method for dynamically modifying the number of slots. Third, OCRA’s relational operator is applied to all possible pairwise object comparisons, an approach that may not be scalable to scenes that contain many objects. In future work, this may be addressed by replacing the transformer component of our model with architectures that are better designed for high-dimensional inputs [16]. Finally, it is unclear how our proposed approach may fare in settings that involve more complex real-world relations, and settings that require the discrimination of multiple relations at once. It may be beneficial in future work to investigate a ‘multi-head’ version of our proposed model (analogous to multi-head attention), in which multiple distinct relations are processed in parallel. A major challenge for future work is to develop models that can match the human capacity for structured visual reasoning in the context of such complex, real-world visual inputs.’
Thank You
Thank you for the response! I appreciate the response that it is the dot product that is playing the key role as evidenced by the failure of the ablation in which slot pairs were concatenated and mapped to single-dimension. Although I find this outcome very intriguing and interesting, I am not fully clear why it does work. Is there something inherent about multiplicative interaction that leads to this? For instance, would the outer product also work similarly? Another question is: Consider that $\mathbf{z}$ contains the appearance information but not spatial position information while $\mathbf{m}$ contains spatial position information but not appearance information. Then, I think that Eq. 4 should lose information about which object appearance is associated with which spatial position. Is this true and is this by design? I also read other reviewers' comments, especially the one asking for the CoRelNet baseline. I think this was a valid point. I agree that this baseline should have been compared. I appreciate the authors' addition of CoRelNet results and I also agree with the authors' response that CoRelNet should struggle to handle permutation invariant representations e.g., slots of slot attention. However, since I missed noticing this related work the first time, I am now less confident that I fully grasp all pieces of the related work. As such, I will reduce my confidence from 4 to 3. But I still maintain my rating of 6 to the best of my understanding.
Thank you for the continued engagement. We have now carried out the analysis that was suggested in question 2 of the initial review (visualization of relational embeddings). We performed this analysis specifically for the same/different task, in the $m=95$ generalization regime. This was the most difficult generalization regime in our task, so we thought this would be the strongest test of whether the relational embeddings truly captured abstract relations. We performed PCA on the relational embeddings (the output of equation 3 in the paper) for a set of 100 problems, and visualized the results along the first two principal components. The results formed two distinct clusters, one for pairs of objects with the same shape, and one for pairs of objects with different shapes. Thus, the relational embeddings identified the relevant relational abstraction (same/different), despite the fact that the inputs involved completely novel shapes not observed during training. We will include these results in the final version of the paper, and thank the reviewer for this suggestion. Inner vs. outer product: This is a great question. To test it, we performed an additional ablation experiment, in which the dot product in the relational operator (equation 3 of the paper) was replaced with an outer product. Note that this results in a matrix of size $D \times D$. We flattened this matrix, and passed it through a learned linear layer to reduce it to size $D$. This version of the model did not perform nearly as well as our primary model (see results in Table below). This suggests that our relational operator depends on *both* multiplicative interaction *and* compression. Ablation models that only have compression (the ablation model with a learned linear projection to one dimension) or only have multiplicative interaction (the new ablation model involving an outer product) do not perform as well. We will add these results and some additional discussion of these issues to the final version of the paper. We thank the reviewer for raising these issues, as we feel that they help to clarify the important factors underlying the implementation of the relational bottleneck. | | RMTS | ID | | -- | -- | -- | | OCRA (original) | **85.31 ± 2.0** | **92.80 ± 0.3** | | OCRA (outer product) | 62.84 ± 1.6 | 69.58 ± 1.1 | Clarification about equation 4: In equation 4, information about the appearance of *individual* objects is discarded, but information about the *relationship* between the appearance of two objects is preserved, and this is then associated with the spatial position of the two objects. The discarding of information about individual object appearance is indeed by design. Please let us know if we can provide further clarification. Regarding the related work, we would be happy to discuss any remaining issues that we can help clarify. We have made our best effort to address all of the concerns raised in the initial reviews and during the discussion period. If there are any issues that haven’t been sufficiently addressed, we would be happy to discuss further.
Thank You!
The results are indeed very interesting. I look forward to seeing the said visualization. I raise my score to 7.
Thank you very much for thoroughly considering my review. All of my questions and suggestions have been adequately responded. I raised the score for Presentation from 2 to 3 and that for Rating from 5 to 6. \# The link to the source code was shared by AC.
Thanks
Thanks to the authors for providing detailed responses to my questions and adding the new experiments, which support the effectiveness of the slot attention. The rebuttal has addressed most of my concerns. I have also read other reviews and increased my rating to weak accept.
Decision
Accept (poster)