Summary
The paper introduces a novel approach for optimizing the application of large language models in resource-intensive vision-language pre-training. Unlike the traditional approaches of using visual features as prompts to guide language models, the paper focuses on identifying optimal prompts to align with visual features. This is achieved through the introduction of the P-Former, a module trained only on linguistic data, eliminating the need for image-text pairings.
Strengths
1. Originality:
The paper demonstrates a high level of originality in its approach to vision-language learning. By introducing the P-Former, the paper presents a unique perspective in contrast to the standard approach of using visual features as prompts. Also, the focus on a modality-agnostic framework expands the applicability to various VL tasks, further enhancing the originality of the paper.
2. Quality:
The paper exhibits a good level of quality in various aspects. The methodology is well-formulated, and the introduction of P-Former is well-justified. Also, it provides sufficient details about the experiments and demonstrates the robustness and flexibility of the framework across different VL tasks.
3. Clarity:
The paper is written with clarity and precision. The abstract, introduction, and related work sections provide a clear context and properly position the paper in the literature. The technical concepts are well-explained, including the P-Former and the decoupled language pre-training approach. The use of figures and tables further help in understanding the experiments and results.
4. Significance:
The significance of the paper is in the decoupled language pre-training approach, with the P-Former's ability to predict ideal prompts, addressing resource-intensive challenges in this field. Also, it offers a fresh perspective on how to leverage large LMs in VL learning scenarios.
Weaknesses
- The idea of the P-Former is interesting, however, it seems to lack an intuitive explanation and motivation. Why learning an ideal language prompt helps more, compared to using visual ones as in the counterpart models?
- There seems to be a lack of some ablations, which naturally arise as questions, for example, experimenting with and without the P-Former module, maybe just by using randomly initialized prompt p, using different sizes/types of language backbones, different ways of initializing the prompt p at the beginning, etc.
- Also, there is a lack of qualitative analysis of the experiments. I would recommend including and analyzing qualitative results in comparison to existing approaches. Presenting visual examples of the model's performance in both successful and failure cases can make the paper stronger.
- Some experiments lack stronger interpretation. I would also encourage the authors to provide more interpretation of some results (e.g. tables 1, 2, 3), rather than describing the tables, to enable the readers to gain better insights into the behavior of the framework.
Questions
- Could the authors provide more details and clearer insights into the P-Former part of the framework and size?
- Across all experiments, it can be seen that more data helps BLIP-2 to outperform the P-Former. I wonder if the P-Former will improve the best performance of BLIP-2 if the pre-training image-text data is larger (129M)?
- In Table 3, it can be observed that the performance of the proposed model is lower when it comes to retrieval. Could you interpret these results and elaborate more on this limited performance?
- What is the reasoning behind the need for a separate pre-training of the P-Former on language data? Why do additional unimodal sentences contribute to the performance in Table 6?
- Since the framework is presented as modality agnostic, do you think it can handle language-only tasks, such as machine translation? If so, which parts of the framework would need to be adapted to handle this?
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Limitations
The authors have provided a sufficient discussion of the limitations of their work in section 5. Due to the usage of pre-trained backbones, I would also encourage the authors to discuss any potential negative societal impact of their work.