Summary
The paper proposes Duty-Distinct Chain-of-Thought (DDCoT) Prompting for multimodal reasoning problems (e.g. VQA). Despite CoT's success in language-only reasoning problems, authors argue that multimodal reasoning challenges CoT as the rationale part is sensitive to the input information. Since image caption is the only source of image information for LLM, once the caption is generated poorly, rationale will intensity hallucinations.
To this end, DDCoT proposes two insights: "keep critical thinking" and "disentangle reasoning and recognition". Specifically, the pipeline is like:
1. Given input question, zero-shot prompt LLM: “please think step-by-step and deconstruct the question down to necessary sub-questions"
2. Determine if each subquestion can be answered without visual information with zero-shot prompt: “Assume that you do not have any information about the picture, try to answer the sub-question and formulate the corresponding sub-answer as ‘Uncertain’ if the sub-question cannot be determined”. This helps deal with hallucination (i.e. LLM makes up facts about images).
3. Use an off-the-shelf VQA model to answer each subquestion "with negative space" (my guess is, each subquestion with 'Uncertain' as answer?)
4. Given subquestions and answers, LLM is prompted to perform CoT reasoning, generating a rationale and a final answer.
The paper also considers a finetuning setup with some architecture inventions.
Results on ScienceQA show
- zero-shot prompting: 1-3% improvement from previous few-shot CoT baselines
- finetuning: more impressive improvements and generalization from baselines.
Analysis on visual information source, rationale generation process, fine-tuning components, and explainability are also conducted to further justify DDCoT.
Strengths
- Multimodal reasoning (e.g. VQA) is an important research direction, and LLM prompting is an emerging and promising approach to it. The paper highlights limitations of existing CoT-based methods for VQA (sensitive to caption; rationale does not improve performance), and propose reasonable solutions to these problems.
- DDCoT proposes an interesting heuristic way to combine LLM and off-the-shelf VQA models as a tool: LLM decomposes questions, decide which parts to call VQA model, and aggregates information back for answer. The approach could be potentially valuable for other tasks (involving video, code, webpages...)
- The consideration of both prompting and finetuning setups add to technical depth.
- Experiments show improvements, and their design looks solid. I appreciate comprehensive and diverse ablations and analysis.
Weaknesses
- The presentation is a bit poor. The concrete problem is just "how to best put visual information in text, so that rationale can help", and the main insight is just "use VQA models across subquestions to best induce visual information, instead of off-the-shelf captioning". But it takes me some time to get it. I don't think "Flexibility/generalizability/explainability" makes too much sense to me --- I don't think few-shot CoT is really labor-intensive (just 3 examples??), inflexible, or hard to explain. The main motivation should be performance and generalization. 3.1.1 seems commonly known and a bit redundant. So intro, 3.1, 3.1.1, 3.1.2 are like four different stories of motivation, and a bit confusing to me. Why not sticking to one single story, instead of writing a story in 4 different ways/views/places?
- Continuing on the presentation issue, I don't get Figure 1,2,3,4,5 --- all are examples, and in fact, Figure 5 should better be the teaser given it actually tells a bit about how the method works. Fig 1's example doesn't really tell much, as MM-CoT and UnifiedQA's rationales are omitted. etc.
- Continuing on the presentation issue, some terms are confusing (or at least lack explaining), e.g. "negative space prompting".
- While finetuning performances seem stronger, prompting performance is only slightly better than few-shot CoT baselines, and there are not a lot of prompting baselines to begin with. In the long run, seems prompting performance will be more important than finetuning performance (one can also imagine multi-modal GPT-4 just solves ScienceQA and beats every model, making it less worthwhile to study...)
- The results are only on ScienceQA, not only how general the findings are, given original CoT is evaluated on various problems, and multimodal CoT should be able to solve a lot of different tasks as well.
Questions
I think the paper can benefit from better writing, and experiments from different datasets.
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Limitations
Appendix E talks about hallucination and social bias.