GraphAdapter: Tuning Vision-Language Models With Dual Knowledge Graph

Summary

This paper introduces GraphAdapter, a novel adapter-style tuning strategy for vision-language models. GraphAdapter can leverage task-specific structure knowledge by explicitly modeling the dual knowledge graph. The authors validate the proposed method on 11 popular benchmarks on few-shot classification setting.

Strengths

- GraphAdapter explicitly models the dual-modality structure knowledge with a dual knowledge graph, allowing for the leveraging of task-specific structure knowledge from both textual and visual modalities. - The authors conducted comprehensive evaluations on 11 datasets and demonstrated the performance under different shots.

Weaknesses

- The authors claimed the previous methods overlook the explicit exploitation of the structure knowledge, but the experimental results showed that the combination of text and visual adapters achieved limited gain (compared to text-only adapter). To some extent, it makes the motivation less convincing. - The knowledge graph is constructed at the very beginning for once, this can be inconvenient in the setting where there will be new data coming sequentially over time, and then the knowledge graph will need to be updated every time. It's important to consider this because the reason we use adapters is we want a quick adaptation with minimal cost. - The performance improvements seem trivial. For example, in Figure 2 and Table 1, most of the improvement over the second-best model is around 1-2%, and sometimes less than 1% or worse. It seems less promising especially considering the complicated design.

Questions

Can the authors also compare the training time/parameters with previous methods to show the efficiency?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

See above.

Summary

This paper proposes a new prompt tuning strategy named GraphAdapter to fuse textual and visual structure knowledge for downstream tasks. It first constructs the dual knowledge graph by taking the textual / visual features of a specific class as nodes and the cosine similarities between these features as edges. After that, the textual feature $z_t$ is warped into the graph space and two GCNs are applied to excavate the knowledge from the textual and visual sub-graphs respectively. Extensive experiments on 11 benchmark datasets show that GraphAdapter consistently outperforms previous works.

Strengths

- The idea of introducing graph learning into prompt tuning methods is novel. - The average performance of GraphAdapter is better than previous methods on few-shot learning, including 1-/2-/4-/8-/16-shots on 11 benchmark datasets. - Ablations also show that GraphAdapter can achieve consistent gains when using different backbones.

Weaknesses

- My major concern is about the **efficiency**. Although modeling all the classes as a knowledge graph is a novel idea, introducing GNNs into prompt tuning are generally time and memory consuming and may violate the original motivation of the *efficient* transfer learning for Vision Language foundation models (VLFMs). In fact, as is mentioned in the appendix (L40), the authors already decouple the original graph of ImageNet (1k nodes) into 4 sub-graphs (256 nodes) to alleviate the computational cost. So the *scalability* of this method maybe limited. - Besides that, one of the most important potential of VLFMs is trasferring to *open-world* scenarios, where the classes are unknown and infinite, which may make the graph impossible to build in advance.

Questions

- In the few-shot training setting, what is the detailed process to obtain the visual features for constructing the visual sub-graph? For example, what is the number of images used here? what kind of data augmentation is applied? - what is the overall cost of GraphAdapter compared to previous methods? For example, (a) The training FLOPs; (b) The training wall-time of one epoch; (c) The inference speed; (d) The number of parameters. - (Minor) The reference figure in L171 is incorrect.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

The limitations are partly addressed. The authors may need to further discuss the limitation of **scalability** as well.

Summary

The paper proposes to utilize graph learning for efficient transfer learning of large vision-language models. The graph learning consists of scene graphs of two different modalities, first one uses the textual features of prompts and the second one uses the visual features of training samples from the downstream tasks. The feature of each prompt aims to align with the the fused graph features during the optimization process. Experiments on 11 few shot classification benchmarks including a few on fine-grained classification tasks show that this method improves the transfer learning performance.

Strengths

The idea of using structured knowledge is intuitive. Using textual and visual both modalities are interesting. The section of different graph adapter variants, text, visual and T-V is worth adding. The ablation experiments of the different coefficients for different modality graphs have added value. The paper proposes an intuitive idea, explains it well and the paper is well written.

Weaknesses

Not a weakness, but a general question. Is there a way to utilize the semantics of visual relationships? The structured knowledge graph in its current form appear to be more of a co-occurrence statistics.

Questions

This is a well written paper. I have one question though: Is there a way to utilize the semantics of visual relationships? The structured knowledge graph in its current form appear to be more of a co-occurrence statistics. e.g. How to distinguish between "person holding a cup" vs "person drinking from a cup" ?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

Broader impacts and limitations are mentioned in the supplementary material, it's the common impact that large vision-language models need to be concerned about - the nature of the pretraining dataset.

Summary

In this paper, the authors present an adapter-style tuning method, termed as GraphAdapter, that explicitly captures the dual-modality structure knowledge by utilizing a dual knowledge graph, leading to enhanced adapter-style transfer learning. Specifically, the authors identify two key challenges in existing adapter-style approaches for efficient transfer learning, including the deficiency in modeling task-specific knowledge from only a single modality, and the neglect of explicitly exploiting the structural knowledge in downstream tasks. Motivated by these two limitations, the authors propose a novel tuning method for visual-language models that incorporates task-specific knowledge for downstream tasks through the integration of textual and visual structure knowledge, based on graph learning. In particular, the proposed method establishes a dual knowledge graph consisting of a textual knowledge subgraph and a visual knowledge subgraph. Consequently, the feature adapter can effectively leverage the inner-modality and cross-modality structure knowledge for superior tuning performance. Experiments on 11 benchmarks convincingly demonstrate the effectiveness of the proposed GraphAdapter approach.

Strengths

1. The proposed GraphAdapter is very well-motivated. The authors have conducted a thorough analysis of the existing ETL approaches, identifying two key limitations, and recognizing the significance of incorporating dual-modality structure knowledge in ETL. Building upon such analysis, the authors develop GraphAdapter, which aims to effectively integrate fused textual and visual structure knowledge using GCN. 2. The experiments are thorough and convincing. The authors perform extensive experiments on 11 few-shot benchmarks, utilizing various backbones such as ResNet-50, ResNet-101, ViT-B/32, and ViT-B/16, as detailed in both the main paper and the supplementary material. In addition, the authors have specifically explored the generalization capability of GraphAdapter on four benchmarks. These experiments convincingly demonstrate the effectiveness of GraphAdapter. 3. The paper is clearly written and easy to follow. The authors extensively elaborate on the details of GraphAdapter, particularly the establishment of the dual knowledge graph. 4. The supplementary material includes extensive implementation details and visualization results. Notably, the authors demonstrate the seamless integration of the proposed GraphAdapter with existing methods such as CaFo and TaskRes*, resulting in consistently improved performance.

Weaknesses

I’m almost satisfied with this paper, with only a few minor concerns as follows. 1. The authors leverage GCN to integrate dual-modality structural knowledge. However, I am interested in understanding the performance of more advanced GNN mechanisms, such as GAT and GraphSAGE, in this context. 2. The authors are suggested to include an analysis of the time complexity, particularly regarding the construction of the textual and visual knowledge subgraphs. 3. Some related works [1, 2, 3] on KG embedding can be included. [1] Wu, Han, et al. "Hierarchical Relational Learning for Few-Shot Knowledge Graph Completion." The Eleventh International Conference on Learning Representations. 2022. [2] Bordes, Antoine, et al. "Translating embeddings for modeling multi-relational data." Advances in neural information processing systems 26 (2013). [3] Xiong, Wenhan, et al. "One-shot relational learning for knowledge graphs." arXiv preprint arXiv:1808.09040 (2018).

Questions

1. Can GCN in GraphAdapter be replaced with other GNNs like GAT and GraphSAGE? 2. What is the time complexity associated with GraphAdapter?

Rating

8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

4 excellent

Presentation

4 excellent

Contribution

4 excellent

Limitations

The limitations have been thoroughly discussed in the main paper.

Thanks for the rebuttal

The authors have addressed most of my concerns. I'm raising my score to 5.

post-rebuttal comments

I thank the authors for posting the rebuttals. I'll keep my original "accept" rating