Localized Symbolic Knowledge Distillation for Visual Commonsense Models

Summary

This paper proposes a framework that can generate global, local, and dynamic descriptions. ChatGPT can generate question-and-answer pairs containing specific regions or descriptive phrases with the proposed tricks. A critic model is trained to filter the generated data. Finally, the localized corpus is used to fine-tune the vision language model.

Strengths

1. This paper proposes a way to acquire localized commonsense knowledge data and enable the vision and language model to answer localized visual commonsense questions. The workload of this work is impressive. 2. The fine-tuned vision language model achieves promising zero-shot performance for three localized visual reasoning tasks.

Weaknesses

1. The proposed framework is more like engineering work. The proposed method to generate localized data pairs is straightforward, and the way to enable the local-level question-answering ability is not novel, limiting its scientific contribution. 2. There are too many tricks, including filtering by critic model and prompts for acquiring question-answering pairs, which decreases the reproducibility and robustness of the whole pipeline. Furthermore, the selection of the vision-language model and large language model, and hyperparameters for each model in each step increase the complexity of the proposed pipeline, and the final vision-language model may be sensitive to the generated data, which makes it hard to iterative improve the final model's performance by improving any step in this pipeline.

Questions

This paper proposes an impressive engineering framework. It would be better to analyze the impact of the quality and amount of the generated data on the final performance of the visual language model.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

3 good

Contribution

3 good

Limitations

I did not find a potential negative societal impact of this work.

Summary

This paper aims to build a instruction following model for **localized** visual commonsense reasoning. The work differs from existing works in that it is able to reason about localized image regions with boxes, without using complicated referring expressions. The paper first collects data by distilling LLMs, i.e., prompting GPT using verbalized image descriptions, which is a concatenation of 3 descriptors: global, local, and QA pairs. Then, the data collected in the first stage is filtered by a critic filter trained using human annotations. Finally, the filtered data is used to train a BLIP-2 model. Experiments are done on 3 datasets for the localized visual reasoning task, showing the performance of the proposed method.

Strengths

- The paper is clearly motivated and the problem studied is well described. - The dataset collection method is well designed. Using (global, local, QA) descriptors to prompt LLM in order to generate question, answer and rationales. - The effectiveness of the critic filtering is well-analyzed and clearly shown in Fig 3. - The paper is well written and easy to follow.

Weaknesses

My major concern is about experiments. The current results may not be very extensive to show the effectiveness of the proposed method. - Box size is an important factor, since the task is localized reasoning. How does the model perform on large and small objects? How does this compare to other methods? First, it would be helpful to describe the distribution of the bbox sizes in the generated data. Second, in the final evaluation, it is also helpful to study the model performance change wrt different box sizes. - three descriptors (global, local, QA) are used to prompt LLMs. Is there a study to show the contribution of each of the three? Since QA is not as widely used as the other two, additional discussions/results showing its effectiveness would be good. - Very limited baselines are compared with. In table 2, only CLIP is compared. In table 3, only ChatGPT. More baselines should be considered. For example, for tab 2, while zero-shot baselines are hard to find, it is still beneficial to show some non zero-shot methods (methods on the leaderboard of these datasets). Although the model performance is not directly comparable, it can give readers some sense of how well the model performs. - (potential extension) While the contribution on the data collection/distillation part is strong, the contribution on the model part is weak. Basically, BLIP-2 is directly used and trained on the collected data. Could there be some model design specific for handling “localized” information? Moreover, the “localized” reasoning in the current paper is constrained to color-coded bbox, while more general forms like free-form masks can be a further extension. - The work is in principle similar to LLaVA, extending it to be “localized”. Could the authors discuss the difference with LLaVA in more detail? - The title suggests “symbolic”, how is this work related to “symbolic”? I feel this is mis-leading. - some typos: L168 QA-MSE, L146 QAR is not defined.

Questions

See weakness.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

N/A.

Summary

In this paper, authors argue that for multimodal LLMs, the previous input formulation is too rigid: either needs to specify the region models should focus on, or brings a verbose object description to refer to the region. A more natural referring expression strategy can help models better understand where the model should pay more attention to and the intent of the input questions. The proposed approach, called Localized Symbolic Knowledge Distillation (LSKD), aims to provide more natural referring expressions. It involves providing literal descriptions of images to a large language model and allowing the model to connect these descriptions with a holistic understanding of the scene. In addition, localized references to specific regions within the image are provided to encourage the model to generate commonsense inferences about those regions. This method effectively distills the large language model's capacity for global and local scene understanding. They demonstrate the effectiveness of this approach by achieving state-of-the-art zero-shot performance on localized visual reasoning benchmarks and conducting human evaluations.

Strengths

Novel research question to distill visual commonsense: Distilling visual commonsense is an interesting question that previous works under-explore. The distilled visual commonsense could provide very useful information for understanding complex situations covered in datasets like VCR, which will make the reasoning more grounded and reliable. Also, this is the first work about automatically acquiring visual commonsense to my best knowledge. Comprehensive experimental results on downstream tasks: I'm very glad to see the distilled knowledge could further help base models on downstream tasks like VCR, Sherlock and VisualCOMET. Also, the small models can even outperform ChatGPT with the help of LSKD when describing the useful information in localized reasoning process. Besides, authors design experiments to examine the effect of steps in LSKD generation methods, which further make the argument stronger.

Weaknesses

Lack of comparison with Sherlock and VisualCOMET: Although the current experiments support that LSKD is helpful for downstream tasks, I'm still a bit confused if LSKD is better than Sherlock and VisualCOMET knowledge bases. These two datasets are both large-scale and of great quality. It's better to compare with competitive resources, instead of just showing that considering external knowledge will help. Comparing VL models and ChatGPT is unfair: Although ChatGPT is prompted with verbalizers, it still can only accept text information, which may ignore many visual information in the images. Especially for datasets like VCR, answering questions may need a very careful observation to some detailed places, which is what VL models like BLIP-2 is better at. Presentation in introduction section is confusing: I don't quite understand the relation between the first and second half of the introduction section. For the first half, I think the authors are trying to say a better referring expression is important. But in the second half (which is the core part of the paper), they just focus on how to distill visual commonsense knowledge. They are a bit connected because both parts mention localized reference, but it has very loose connection with discussion about the quality of referring expression.

Questions

1. Is there any way to apply this method to the datasets with no bounding box annotations? That will further improve the scalability of this method. 2. Is there any study to explore the correlation between the ration of error training instances and the downstream task performance? 3. Could you also apply LSKD to models like CLIP? It seems that in Table 2, they just use BLIP-2 ViT-G to show the effectiveness of LSKD. 4. More qualitative error analysis is welcome in the modified version.

Rating

8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The limitations about introducing errors in the intermediate steps are reasonable.

Summary

The paper introduces Localized Visual Commonsense models that enhance vision-language (VL) models by allowing users to specify specific regions within images. The authors train their model by collecting commonsense knowledge from a large language model (LLM) using global literal image descriptions and automatically generating local literal region descriptions from VL models. They also use a critic model to select high-quality examples. The results show that training on this localized commonsense corpus improves the performance of VL models of passing generated referring expressions to an LLM.

Strengths

+ The paper is well-motivated. The introduction of Localized Visual Commonsense models represents a novel solution to build a more general multi-modal model. + The constructed data is of large scale, which contains 1M samples over 250K images. + The extensive experiments show that the proposed dataset can support model to perform better zero-shot inference.

Weaknesses

- How does this Dataset reflect knowledge? Since ChatGPT's input is based on factual descriptions of images, how can we ensure the questions generated by ChatGPT are related to external knowledge? On the other hand, how do the authors define what are knowledge-related questions? - For some knowledge-related questions, it seems that ChatGPT needs to guess the state of the subject in the image. Could this process result in hallucination issues? I notice that there's a critic filtering in the method, which rates the results by training on annotated data. What types of patterns is this model expected to learn, and under what circumstances does it consider the generated data to be inadequate? From Section 2.3, it seems to be judging visual correctness, but how can the correctness of the generated knowledge be determined? For example, in Fig1, the model assumes that [0] and [2] are teaching [1]. This is a guess made by ChatGPT, as no similar descriptions are generated by the visual models, indicating that they cannot capture this information. So how would the critic model determine if this kind of guess is based on reality or is just a hallucination?

Questions

My major concern with this work is how to guarantee the dataset quality since there are a lot of automatic generation methods in dataset construction. So I would like to hear more illustrations about that.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

N/A

Summary

This work presents a localized visual common sense model that can support region-level inputs for knowledge reasoning. Specifically, the proposed methods prompts large language models to collect commonsense knowledge from global image descriptions and local descriptions. A critic classifier trained over a small number of annotation data is used to select high-quality examples. The distilled localized common-sense corpus can be further used for fine-tuning vision-language models. The experimental results demonstrate improvement over baselines in the zero-shot setting.

Strengths

a). Strong motivation; supporting region-level references and reasoning is important for many applications. b). Technical sound; provide detailed and clear explanation of each component; experiments and ablation studies are comprehensive. c). Thorough evaluation; have human evaluations comparing zero-shot performance of localized reasoning with generative models.

Weaknesses

a). A new way of distilling vision-language knowledge but no framework-level innovation compared with text only symbolic distillation. b). The effectiveness of supervised critic: as the validity of instance annotation is only 45% and the trained classifier can only achieve 0.66 on F1 score. It is unclear whether the filtering process can be effective in selecting high-quality examples.

Questions

a). Can the collected 1M instances of localized commonsense knowledge be used for other relevant vision-language tasks? Will the knowledge further benefit those tasks? b). In the Section of “effect of filtering”, the comparison with LLaVa datasets seems unfair since the number of LLaVa is fixed. What is the number of training data for LLaVa?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

n.a.

Thanks for your response -- AC Comment

Dear Authors, Thanks for providing your responses to the reviewers' comments. They are comprehensive and answer many of the concerns raised during the initial review phase. The discussion with other reviewers also provides additional context for the questions raised during the initial review. While there is still time in the discussion phase, the reviewers can engage for further clarifications. -- Your AC