Provable Compositional Generalization for Object-Centric Learning

Dear Reviewer, We thank you for your positive review and your valuable feedback. We were delighted to see that you found our theory “sound and significant”, that you found our proposed framework “elegant”, and that you found our manuscript to be “very well written”. We address your comments below. ___ **Comment:** ”How does the effect of the consistency loss scale with the number of slots?” **Response**: Thank you for asking this interesting question! In App. C.3, we now include an experiment that ablates the impact of the number of slots on the consistency loss. As the number of slots increases from 2 to 3 and 4, optimization of the consistency loss becomes more challenging, and its value grows. This is not unexpected as the number of possible latent slot combinations grows combinatorially with the number of slots, making sampling the space more challenging. Furthermore, as the consistency loss increases, we see worse performance on OOD metrics, as our theory suggests. We now include a discussion of potential solutions for scaling this loss to more complex settings in App. B.5. ___ **Comment:** “What is the impact of how slot-supported the training data is? What is the impact of the width of the blue band on empirical effectivity?" **Response**: Thank you for raising this interesting point! We address this question by examining two questions empirically: 1. What is the effect of the width of the diagonal band in latent space, i.e., the effect of the training data's size (and diversity)? 2. What happens if slot-supportedness is not fulfilled? We address the first point in App. C.1 and show that a model trained with the consistency loss yields high OOD performance across different widths. On the other hand, a model trained without consistency loss requires a much wider band to reach the same OOD performance. We address the second point in App. C.2 by creating a dataset in which there is a gap in the latent space such that it is no longer a slot-supported subset. In this case, we see that OOD reconstruction increases sharply in the region around this gap. ___ **Comment:** ”How does the method hold up on non-synthetic data?” **Response**: We agree with the reviewer that understanding our results on non-synthetic data is important; however, this is challenging for two main reasons. Firstly, for real-world data, one does not generally have access to the ground-truth latents making our evaluation schemes inapplicable. Secondly, even if access to ground-truth latent information is available, our experiments require being able to sample latents densely from a slot-supported subset. Specifically, our experiments rely on sampling from a diagonal strip in the latent space with a small width. If such a region were sub-sampled from an existing dataset, this would leave a tiny number of data points that are insufficient to train a model. To this end, our experiments require access to a dataset's ground-truth renderer so latents can be sampled densely. This is not available in most cases, however. We now discuss this in a paragraph in App. B.6. ___ **Comment:** “What if you … allow for a low-expressivity non-linear combination at the end for rendering?” **Response**: Regarding a “low-expressivity non-linear combination at the end for rendering”, we note that the softmax in the alpha-masking of Slot Attention in our experiments in Sec. 6.2 can be understood as just this. For these experiments, the model optimizes the consistency loss and includes a deterministic encoder, thus matching all theoretical assumptions except for _additivity_. In these cases, we found that the nonlinear combination leads to a steep increase in the isolated decoder OOD reconstruction error (See Fig. 5, left). Developing theory to allow for such nonlinear combinations is an important direction for modeling real-world data. We view our work as a crucial initial step in this direction.

Dear Reviewer, We thank you for your positive review and your valuable feedback. We are happy to hear that you “appreciated” our theoretical analysis, that you found our work “well-written” and “easy to read”, and that you found multiple aspects of our work “interesting and inspiring”. We address your comments below: ___ **Comment:** ”It would be great if the assumptions could be relaxed.” **Response**: We agree with the reviewer that relaxing our theoretical assumptions, namely _additivity_ and _compositionality_, is essential for modeling realistic data to allow for more complex interactions between slots. We aimed to highlight this in our discussion paragraph, “Extensions of Theory”. While interactions between slots cannot be arbitrary, it is likely that further degrees of interaction may be allowed beyond additive interactions. We view this as an important direction for future work and believe that the current work constitutes a crucial step towards a more general result. ___ **Comment:** "Are there results in more complex environments?" **Response**: Understanding our theoretical results for more complex data is indeed important for better understanding its applicability. This is challenging for real-world datasets, however, as one does not generally have access to the ground-truth latents making our evaluation schemes inapplicable. Moreover, using more complex synthetic datasets typically used in object-centric learning is also not straightforward for our experiments due to the need to sample latents densely from a slot-supported subset. Specifically, our experiments rely on sampling from a diagonal strip in the latent space with a small width. If such a region were sub-sampled from an existing dataset, this would leave only a tiny number of data points that are insufficient to train a model. To this end, our experiments rely on access to the ground-truth renderer for a dataset such that latents can be sampled densely; however, this is not available in most cases. We now discuss this point in a paragraph in App. B.6.

Dear Reviewer, We thank you for your review and your valuable feedback. We are happy that you found our work makes “theoretical progress on understanding compositional generalization in object-centric representation learning“. We address your comments below: **Comment:** “Compositionality and irreducibility are quite restrictive.” **Response**: Regarding compositionality, we agree with the reviewer about the restrictiveness of this assumption. Compositional decoders are special cases of additive decoders (see App. A.7) and thus share the same limitations: They cannot perfectly model objects with complex interactions such as occlusion or reflection. Irreducibility, on the other hand, roughly states that parts of the same object share information. We argue that this assumption is a more fundamental property of objects and will thus be valid more generally for multi-object scenes. **Comment:** “Additive decoder limits modeling of complex object interactions and relations.” **Response**: As noted in our “Extensions of Theory” paragraph in the Discussion section, we agree with the reviewer that additivity is indeed insufficient for modeling more complex object interactions. We believe, however, that this assumption can be relaxed to allow for more complex interactions and view this as an important direction for future work. **Comment:** ”Consistency regularization requires sampling implausible object combinations. More principled schemes could improve results.” **Response**: As noted in our “Extensions of Experiments” paragraph in the Discussion section, we agree regarding the limitations of our consistency loss implementation and approaches for improving it. One such approach could be to learn a prior over latent slots and sample slot combinations more rigorously according to their likelihood under the prior. This could avoid sampling implausible combinations; however, such a scheme is challenging as it relies on the likelihood of being valid for OOD combinations of slots. Another possibility would be to include heuristics to directly filter combinations based on prior knowledge of the data-generation process, e.g., to filter objects with similar coordinates that would intersect. We believe that further explorations in such directions are a promising and important direction for future work to make this method practical for more complex datasets. We now include this discussion in App. B.5. **Comment:** “Experiments only validate the theory on simple synthetic datasets.” **Response**: We agree with the reviewer that understanding our theoretical results on more diverse and real-world data is important for better understanding its applicability. This is indeed challenging for real-world datasets, however, as one does not generally have access to the ground-truth latents making our evaluation schemes inapplicable. Moreover, using the synthetic datasets typically used in object-centric learning is also not straightforward for our experiments due to the need to sample latents densely from a slot-supported subset. Specifically, our experiments rely on sampling from a diagonal strip in the latent space with a small width. If such a region were sub-sampled from an existing dataset, this would leave only a tiny number of data points that are insufficient to train a model. To this end, our experiments rely on access to the ground-truth renderer for a dataset such that latents can be sampled densely; however, this is not available in most cases. We now discuss this paragraph in App. B.6. **Comment:** “The proposed methods might pose scalability issues for very large datasets or more complex models.” **Response**: Thank you for mentioning this. Methods based on latent slots have in fact been shown to scale to more complex settings (e.g., [1, 2]); however, we agree regarding the scalability of the consistency loss, which we aimed to highlight in our “limitations of experiments” paragraph. We have included a figure in App. C.3 highlighting the challenges in scaling this loss as the number of objects increases. Regarding computational costs, the loss requires additional passes through the encoder and decoder as well as computation of the encoder’s gradients wrt. the loss. We found this to increase training time by a maximum of 28% across runs. We have incorporated a discussion of this point in App. B.5. We hope these clarifications and revisions are sufficient for the Reviewer to confidently increase their score. Furthermore, we noticed the Reviewer left a score of “2” regarding the soundness of our manuscript. We found this noteworthy as we are not aware of any inaccuracies in our theoretical or empirical contribution. Thus, if the soundness score was the result of one of the points raised by the reviewer above, we hope our comments were sufficient to resolve the reviewer's doubts. [1] https://arxiv.org/abs/2206.07764 [2] https://arxiv.org/abs/2205.14065