Learning to Reason Iteratively and Parallelly for Complex Visual Reasoning Scenarios

Summary

he paper addresses visual reasoning problems. It introduces a fully neural iterative and parallel reasoning mechanism (IPRM) that combines iterative computation with the ability to perform distinct operations simultaneously. Evaluation is performed on visual reasoning datasets such as CLVER, AGQA, CLEVR-Humans and STAR dataset.

Strengths

* The proposed method is novel and interesting. * The paper is well written and easy to understand. * The paper includes in-depth analysis of the proposed method. * The method shows promising zero-shot performance on CLEVRER-Humans,

Weaknesses

* Generalization Results: The method shows zero-shot results only on CLEVRER-Humans. This is in contract to current SOTA video-language models such as LLaVA-Next or LLaMA-Vid which show zero-shot results across multiple datasets such as MSRVTT to TGIF-QA. Can the proposed model architecture be scaled to create more general purpose models similar to current video-language models? * Multi-turn QA (related to the above point): The standard transformer architecture allows for multi-turn dialogue, where the user can ask the model multiple questions about a video (e.g. LLaVA-Next or GPT-4V). Can the proposed model architecture be extended to allow for multi-turn dialogue? * Results on STAR: Recent works such as "Look, Remember and Reason: Grounded reasoning in videos with language models, ICLR 2024" outperform the proposed model. * Results on GQA: “Coarse-to-Fine Reasoning for Visual Question Answering, CVPR Workshops 2021” achieves 72.1% compared to 60.5% reported for IPRM. (Note that the use of scene graphs during training should not be a reason not to compare, instead the paper should discuss approaches to close the performance gap). * The reasoning steps in Figure 7 are not very interpretable. In Figure 7 (left) the model does not seem to attend to the large cylinders in the first step. Similarly, in the Figure 7 (middle) the bottom reasoning chain seems to empty steps where the model does not look at any object. It would also be interesting to see the reasoning chains in case of real-world data such as STAR.

Questions

* Can the proposed architecture be scaled to provide zero-shot results on multiple datasets similar to current SOTA video-language models? * Can the proposed architecture handle multi-turn conversations? * The paper should compare/discuss SOTA approaches such as "Look, Remember and Reason: Grounded reasoning in videos with language models, ICLR 2024" on STAR and “Coarse-to-Fine Reasoning for Visual Question Answering, CVPR Workshops 2021” on GQA.

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

Yes.

Summary

This paper introduces the Iterative and Parallel Reasoning Mechanism (IPRM), a novel neural architecture designed to enhance visual question answering (VQA) by combining iterative and parallel computation. The IPRM aims to address the limitations of existing methods that rely solely on either iterative or parallel processing. The iterative component allows for step-by-step reasoning essential for complex queries, while the parallel component facilitates the simultaneous processing of independent operations, improving efficiency and robustness. The authors propose a lightweight, fully differentiable module that can be integrated into both transformer and non-transformer vision-language backbones. The IPRM outperforms current state-of-the-art methods across several benchmarks, including AGQA, STAR, CLEVR-Humans, and CLEVRER-Humans, demonstrating its capability to handle various complex reasoning tasks. Additionally, the mechanism’s internal computations can be visualized, enhancing interpretability and error diagnosis.

Strengths

1. The combination of iterative and parallel reasoning within a single framework is novel, addressing specific limitations of existing VQA methods. 2. The IPRM demonstrates superior performance across multiple challenging VQA benchmarks, demonstrated by extensive experimental results. 3. The paper is generally well-structured.

Weaknesses

1. While the paper is well-structured, certain sections, particularly those describing the technical details of the IPRM, can be dense and challenging to follow. 2. The paper could benefit from a more detailed discussion on the scalability of the IPRM, particularly in terms of computational resources and training time.

Questions

1. Can the authors provide more insights into the scalability of IPRM? Specifically, how does the computational complexity of IPRM compare to other state-of-the-art methods? 2. It would be helpful to include an analysis of the sensitivity of IPRM’s performance to its hyperparameters, such as the number of parallel operations and the length of the reasoning steps.

Rating

6

Confidence

4

Soundness

4

Presentation

3

Contribution

4

Limitations

The authors acknowledge the limitations of their work, particularly regarding the potential scalability challenges and the need for further evaluation on diverse datasets.

Summary

The paper proposes an iterative and parallel reasoning mechanism (IPRM) for VQA tasks, combining step by step iterative computation with parallel processing of independent operations, and also maintaining an internal memory of operation states and results states. Firstly at each reasoning step, previous operation states attend to language features to form latent operations, then latent results are formed by attending to visual features based on the newly formed latent operations and prior result states. Finally new memory state is computed by combining the latent operation and result states alongwith prior memory states with a lookback window. IPRM outperforms previous methods on several complex image and video question answering tasks. IPRM is also lightweight and sample efficient. Ablation experiments demonstrate the importance of different components.

Strengths

1. The paper is well written and easy to follow. 2. A novel way of combining iterative and parallel processing for VQA tasks. 3. The approach demonstrates decent improvements on three video (STAR, AGQAv2, CLEVERER-Humans) and three image question answering tasks (CLVER-Humans, CLEVR-CoGenT, and GQA), alongwith the capability to generalize zeroshot and in out of domain settings (CLEVR-CoGenT). 4. The approach also requires fewer training samples and parameters.

Weaknesses

No major weaknesses, please see questions below.

Questions

1. What is the reason behind applying a MLP with a nonlinear activation instead of just linearly transforming prior operations to get query in Eq 4, as is done in standard attention mechanisms? 2. Can the authors perform some ablations on operation execution component, regarding the importance of using previous result states and newly formed operations in forming attention key? 3. Based on the ablation plots, figure 6 (iii) it seems r=1 performs better than r=2, then why was r=2 chosen for the experiments? 4. For the window length ablation it would be informative to perform the ablation for W>3, to make sure the performance doesn’t increase further with window length. Also it seems W=3 performs slightly better than W=2? 5. How does IPRM do on the CLOSURE [1] dataset (both zeroshot and finetuned)? Suggestions: 1. It would be informative to also include the MAC model in the comparison of the parameters and sample efficiency in Figure 5. 2. Some of the recent prior work on visual reasoning [2,3,4] haven’t been cited. 3. The authors should include a discussion of limitations/future directions in the main paper. [1] - Bahdanau, D., de Vries, H., O'Donnell, T.J., Murty, S., Beaudoin, P., Bengio, Y. and Courville, A., 2019. Closure: Assessing systematic generalization of clevr models. arXiv preprint arXiv:1912.05783. [2] - Mondal, S.S., Webb, T. and Cohen, J.D., 2023. Learning to reason over visual objects. arXiv preprint arXiv:2303.02260. [3] - Webb, T., Mondal, S.S. and Cohen, J.D., 2024. Systematic visual reasoning through object-centric relational abstraction. Advances in Neural Information Processing Systems, 36. [4] - Mondal, S.S., Cohen, J.D. and Webb, T.W., 2024. Slot abstractors: Toward scalable abstract visual reasoning. arXiv preprint arXiv:2403.03458.

Rating

7

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors have discussed the limitations in the appendix.

Summary

This paper introduces the Iterative and Parallel Reasoning Mechanism (IPRM), a neural architecture for complex visual reasoning and question answering tasks. IPRM involves a three-step iterative process: Operation Formation, Operation Execution, and Operation Composition. In Operation Formation, IPRM generates new parallel operations by extracting relevant language information based on previous operation states. During Operation Execution, these new operations are processed simultaneously, retrieving pertinent visual information guided by both newly formed operations and existing result states. Finally, in Operation Composition, IPRM integrates the new operations and their results into memory by dynamically combining them with each other and with previous operation states. Experiments are performed on video reasoning like STAR, AGQAv2, visual reasoning datasets like GQA, CLEVR.

Strengths

- The motivation behind the architecture is interesting and sound. - The experiments show improved results comparing with other methods on several benchmarks. - The visualization is interesting and somehow can explain intuition of the model.

Weaknesses

• Some notations need further clarifications: The paper needs to improve consistency and clarity in its notation. For example, it is unclear if K in Figure 2 and its caption is equivalent to N_op used elsewhere. Some variables in equations are not properly defined or explained, such as U and H in equation (9), and u2 and v2 in equations (15) and (16). • Model scaling: How does the performance change if the number of parameters is increased/decreased? While the authors claim IPRM can be applied to different vision and language backbones, they don't provide evidence of its performance when scaling up. This omission leaves questions about the model's scalability and efficiency unanswered. • What is the intuition behind using fixed numbers of parallel operators and computation steps, given that different questions likely require different numbers of operations and reasoning steps? Is there any potential drawback of using fixed parameters in this context? • Can the author provide some analysis about failure cases of model?

Questions

Please see weaknesses.

Rating

5

Confidence

4

Soundness

3

Presentation

2

Contribution

3

Limitations

N/A

Official comment by Reviewer nzfV

Thank you for the detailed rebuttal and the additional experiments. Can the authors do some failure mode analysis and provide some intuition why IPRM doesn't perform so well on the or_mat and compare_mat settings of the CLOSURE dataset?

Great work!

The rebuttal has addressed most of my concerns. I will raise my score.