Does Visual Pretraining Help End-to-End Reasoning?

Summary

- The authors explore whether self-supervised visual pretraining can allow end-to-end methods to be more general purpose. - The paper proposes a self-supervised representation learning framework for this task, based on slot tokens, and evaluates performance on multiple benchmarks.

Strengths

- The paper is well-written, studies an interesting problem, and proposes a satisfying solution. - The idea of an implicit visual concept learner with self-supervised pre-training, that retains the same benefits as fully symbolic methods, is compelling and will be of interest to the neuro-symbolic & end-to-end learning communities.

Weaknesses

- W1. A key weakness of the paper is that the evaluation datasets are based on CLEVR objects in *synthetic* images. I would like to see experiments on real-world images; I’m not yet convinced that this self-supervised objective with slot tokens would work in natural images that may contain clutter, etc. I am happy to raise my score if authors add evaluation on a real-world dataset. - W2. I am interested in the performance of using slot-based inputs, instead of explicitly object-centric representations, for neuro-symbolic methods. The proposed methods leverages these softer slot tokens + end-to-end methods; would the change from object-centric input to slot tokens decrease performance of neuro-symbolic methods significantly? Comparison to the NS-CL [1], a neuro-symbolic approach, with slot tokens may be interesting. [1] The Neuro-Symbolic Concept Learner, ICLR 2019

Questions

See above ^

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

Yes

Summary

The authors investigate whether unsupervised pre-training yields representations that are suitable for solving reasoning tasks. To that end, they propose a new transformer-based model inspired by masked autoencoders that is pre-trained on CATER. The authors find that the proposed model outperforms supervised pre-training counterparts and it is competitive with respect to object-centric approaches. Interestingly, the representations learned on CATER transfer well to RAVEN, achieving state-of-the-art results. In addition to the main experiments, the authors include multiple ablations and visualizations of the learned slots. Additional experimental details, ablations on the number of slot tokens, visualizations, and pre-training data are provided in the Appendix.

Strengths

Originality ======== * The proposed pre-training scheme is novel to the best of my knowledge * The finding that representations learned on CATER transfer to RAVEN is novel and interesting Quality ====== * The proposed method is technically sound * The model design choices are properly justified with a good number of ablations (6 tables in the main text and one in the appendix). * Besides comparing with state of the art methods, the authors provide additional baselines such as supervised pretraining, leading to the finding that pre-training on JFT is not significantly better than pre-training on ImageNet. * The slot visualizations provide additional insights on the inner mechanisms of the learned model. Clarity ===== * The text is well-written and easy to read. Significance ========== * The fact that unsupervised pre-training on CATER results in representations that compete with dedicated slot-attention methods and transfer to RAVEN matrices is an important finding that will have an impact in the research community

Weaknesses

Originality ======== * Unsupervised pre-training for solving reasoning tasks has already been explored in the past (slot attention) and the proposed method is close to existing methods like MAE. However, I think the originality of this work lies on the empirical evidence obtained from training the proposed method in different ways applied to reasoning tasks. Quality ====== * The authors make a clear distinction between object-centric two-stage methods using e.g. slot attention, and their proposed approach. However, isn’t the proposed approach also slot-based and two-stage (pre-training + fine-tuning)? Clarity ===== * Figure 1 shows a difference between existing slot attention methods (b) and the proposed method (c) in the fact that the proposed method is single-stage. However, does not the method have a pre-training and fine-tuning stage? * In Figure 2, the slot tokens suddenly appear at the output of the encoders, should not they be in the input too (like the [CLS] tokens)? Minor ===== Line 173: reasonign

Questions

* Could you clarify the conceptual difference between unsupervised slot attention methods and the proposed method (see clarity above)? * Is [A] relevant to your work? [A] Faulkner, Ryan, and Daniel Zoran. "Solving Reasoning Tasks with a Slot Transformer."

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The authors include discussion on the limitations in the last section.

Summary

This paper explores with the need of explicit visual abstraction for visual reasoning tasks.Instead, this paper explores the use of implicit representations learned through a slot attention module.The proposed approach uses slot attention in a frame prediction frame-work that aims to compress visual input information to set of representative tokens/slots. The method is evaluated on CATER, ACRE and RAVEN tasks.

Strengths

* The paper presents an interesting method that seems to generalize across a diverse set of datasets. * The results on RAVEN are very promising. * The paper is well written.

Weaknesses

* The proposed method has lot of similarities to prior work: "Learning What and Where: Disentangling Location and Identity Tracking Without Supervision, ICLR 2023", which also uses slot attention to learn implicit representations. * The above mentioned work achieves 90.7% Top-1 accuracy on CATER, significantly better than the 70.14% reported in this work. * Therefore, a detained discussion about the similarity and differences to prior work is necessary. * The proposed method seems to be designed for image sequences. Could the proposed implicit representation be extended to visual reasoning problems which take a single image as input e.g. CLEVR? * Important details about the evaluation protocol is unclear:: for evaluation on CATER in L242 it is stated that "100 uniformly sampled frames of stride 3 during evaluation". Why is the full sequence not utilized during evaluation? * The paper does not report results on the moving camera split of CATER. Why is this the case? Is the method limited to only static camera sequences?

Questions

* An clear explanation of the advantages/limitations with respect to "Learning What and Where: Disentangling Location and Identity Tracking Without Supervision, ICLR 2023", would be greatly appreciated. * Details of the evaluation protocol on CATER should be discussed in more detail. * Details of the applicability of the proposed method to the moving camera split of CATER should be discussed in more detail.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

Currently, the limitations are not discussed is detail. While evaluation on synthetic data is definitely a limitation, an analysis of failure cases would be helpful.Furthermore, broader societal impacts are also not discussed.

Summary

This paper investigates if an end-to-end trained neural network is able to solve challenging visual reasoning tasks. A new representation learning framework named implicit visual concept learner (IV-CL) is proposed. It consists of two main components. First, a single image is compressed into a small set of tokens with a neural network. Second, the slot tokens are provided as context information for other images in the same video, where the goal is to perform video reconstruction via the masked autoencoding objective with the temporal context. The method outperforms traditional supervised pretraining works, and several observations are summarized based on experiment results.

Strengths

The paper is with good motivation and the analysis of current problems of visual reasoning is reasonable and insightful. Different from explicit representations, implicit representations may correspond to part of, or a combination of human-interpretable symbols, or may be discretized and adjusted based on human understanding or model feedback. It has several advantages: free from a pre-defined concept vocabulary or concept classifiers, and avoids early loss of information. Thus, the paper proposes a new framework named implicit visual concept learner (IV-CL) and summarizes several observations based on experiment results.

Weaknesses

The proposed implicit visual concept learner (IV-CL) is not a sound solution to the motivation of “implicit representation” of the “explicit” reasoning process, because the model based on self-supervised pretraining is totally an implicit one. The image is compressed into a small set of tokens, the slot tokens are provided as context information via a temporal transformer network for other images in the same video, and the learning process is supervised by masked autoencoding and video reconstruction. The explicit information adopted in previous works (e.g., [1,2]) is abandoned. The information includes localized objects, object attributes (e.g., color, shape), and spatial relationships. Reasonable implicit reasoning should imply the information. For example, the visual patches attention for each slot is in correspondence with one piece of information. The visualization in Fig.3 shows some interpretable results obtained from the self-supervised training. However, without any supervision, it is not controllable and stable, and difficult to generalize to more complex visual reasoning applications. Therefore, method design should be reflected and refined to match the goal proposed in the introduction (L36-39): “Some of the learned implicit representations may be discretized into human-interpretable symbols for the purposes of human understanding of and feedback to the model. Others may correspond to part of or a combination of human-interpretable symbols.” Alternatively, it is also reasonable to study the effect of pretraining on visual reasoning. However: 1. The introduction starts with the problem of existing explicit reasoning (L33-44). If the core contribution is a new pre-training paradigm, the demonstration should be adjusted. 2. More importantly, the paper is built upon small, simplified synthetic data, which have a huge gap with practical visual reasoning applications, e.g., VQA in real scenes with diverse visual concepts. Thus, the result and performance of pre-training on these benchmarks are not that convincing and lack generalization. The limitation is discussed in the paper, but I regard it as a crucial problem that should be considered in this paper. [1] Mao, J., Gan, C., Kohli, P., Tenenbaum, J. B., & Wu, J. (2019). The neuro-symbolic concept learner: Interpreting scenes, words, and sentences from natural supervision. arXiv preprint arXiv:1904.1258 [2] Yi, K., Wu, J., Gan, C., Torralba, A., Kohli, P., & Tenenbaum, J. (2018). Neural-symbolic vqa: Disentangling reasoning from vision and language understanding. Advances in neural information processing systems, 31.

Questions

Discuss the question in the “Weakness” section: Why IV-CL learns "implicit" "concept"? Or why does the pre-training on current datasets make sense?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

The paper has discussed realistic limitations. However, an important limitation should have been considered (please refer to “Weakness” section).

Update

CATER moving camera results improves the paper, will update the rating. However, note that very recent works clearly outperform the proposed approach by a large margin: Look, Remember and Reason: Visual Reasoning with Grounded Rationales, KLR@ICML 2023.

Response to rebuttal

Thanks for the clarifications. I am raising my score due to experiments on the Something-Else benchmark. I would recommend moving these results to the main text, and adding more baselines for comparison.

Official Comment by the Authors

Dear reviewers and area chairs, As the reviewer-author discussion phase is coming to an end, we would like to say thank you, and that we truly appreciate your constructive feedback, which helps substantially improve our submission. We are glad that our rebuttal and the additional experimental results have addressed the concerns from all reviewers, and we will incorporate the suggested changes (e.g. moving Something-Else results to the main text, clarifying our definition of "implicit concepts", discussing additional related works) in the final version. Best, The Authors