TransAgent: Transfer Vision-Language Foundation Models with Heterogeneous Agent Collaboration

Summary

This paper proposes the TransAgent framework, which unifies and transports knowledge from isolated agents to guide CLIP in generalizing through multi-source knowledge distillation. This framework allows flexible collaboration with 11 heterogeneous agents to enhance vision-language foundation models. Importantly, this approach incurs no additional cost during the inference phase.

Strengths

**[Interesting idea]** Leveraging heterogeneous agent collaboration for generalized vision language models is intriguing. **[Good presentation]** The paper is well-written and organized, which is easy to follow. **[Well-illustrated figures]** The figures shown in this paper are clear enough to tell the workflow of the method.

Weaknesses

**[Overclaimed statements]** The paper mentions that “it is the first unified transfer framework for generalizing vision-language foundation models with heterogeneous agent collaboration.” However, CaFo [67] had done similar things by adopting multiple heterogeneous agents. This statement may need a revision to differentiate from CaFo. **[Some SOTA methods are not compared]** Some works to be compared or discussed in related works are listed as follow: “PromptKD: Unsupervised Prompt Distillation for Vision-Language Models, CVPR2024”, “DePT: Decoupled Prompt Tuning, CVPR2024”, “Beyond Sole Strength: Customized Ensembles for Generalized Vision-Language Models, ICML2024”, “Consistency-guided Prompt Learning for Vision-Language Models, ICLR2024”, “GraphAdapter: Tuning Vision-Language Models with Dual Knowledge Graph, NeurIPS2023” and “Task Residual for Tuning Vision-Language Models, CVPR2023”. **[Need further explorations]** The knowledge distillation technique used in this work is too straightforward. It would be nice to see an advanced distillation with heterogeneous agents. **[Inadequate experiments]** There are some experiments to be done for verifying the effectiveness of the proposed method. (i) Training time comparisons with other methods and more comparisons for inference time in Table 10; (ii) It would be nice to see the performance by increasing each agent gradually. This can help better understand which agent is more effective and which one is useless. (iii) Ablation study conducted using VAC, LAC, or MAC individually, as well as in combinations of any two. **[Disorganized reference format]** Please reformat the references, e.g., “Conference on Computer Vision and Pattern Recognition”, “Proceedings of the IEEE/CVF conference on computer vision and pattern recognition” and “Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)”.

Questions

Please refer to the weakness section.

Rating

6

Confidence

5

Soundness

2

Presentation

3

Contribution

3

Limitations

Yes, they have provided the limitation section.

Summary

The paper introduces TransAgent, a novel framework designed to enhance vision-language foundation models like CLIP through the integration of knowledge from diverse, pre-trained expert models. These experts, which include vision, language, and multi-modal models, possess rich knowledge acquired from different modalities, tasks, networks, and datasets. TransAgent addresses the challenge of generalizing these models to new domains, especially under low-shot conditions, by proposing a unified transfer framework that leverages heterogeneous agent collaboration. The framework employs a mixture-of-agents gating mechanism to adaptively integrate external knowledge and uses multi-source distillation to transfer this knowledge into CLIP, enhancing its generalization ability without additional inference cost. Experiments demonstrate TransAgent's state-of-the-art performance on 11 visual recognition datasets, significantly outperforming methods like CoOp under low-shot settings and showing remarkable results on EuroSAT with large domain shifts. The paper's contributions lie in its innovative approach to knowledge transfer, the flexibility of its framework, and its significant improvements in model generalizability and efficiency.

Strengths

The paper presents a highly original approach to enhancing vision-language foundation models by proposing the TransAgent framework. This framework innovatively integrates knowledge from a diverse set of pre-trained expert models into a unified system. The use of a mixture-of-agents gating mechanism and multi-source distillation is a unique contribution that sets this work apart from existing methods. The authors have conducted extensive experiments on 11 visual recognition datasets, demonstrating the effectiveness of TransAgent under various low-shot scenarios. The comparison with state-of-the-art methods like CoOp and the detailed ablation studies provide a solid understanding of the framework's capabilities and the impact of different components. The paper is well-structured, with a clear abstract and introduction that succinctly summarize the contributions and scope of the work.

Weaknesses

After changing the expert agent selection, the model needs to be retrained, and this structure is not plug-and-play. The use of integrated features from multiple teacher models as supervision for distillation is widely used in other fields, and the authors have transferred this approach to the domain of this paper, which lacks sufficient novelty

Questions

Could the authors provide insights into how the TransAgent framework generalizes to domains outside of the tested benchmarks, especially those with significantly different characteristics? The paper mentions the potential introduction of irrelevant knowledge during distillation. What strategies are in place or could be considered to mitigate this issue? The role of learnable prompts seems pivotal. How were the prompts engineered, and what impact did they have on the model's performance?

Rating

5

Confidence

2

Soundness

3

Presentation

3

Contribution

2

Limitations

The authors have adequately addressed the limitations.

Summary

The paper focuses on the challenge of vision-language foundation models (e.g., CLIP) struggling to generalize to diverse target domain data in downstream tasks. It highlights the potential of using expert models, which are pre-trained on various modalities, tasks, networks, and datasets, to improve generalization. The proposed TransAgent framework integrates the knowledge of these isolated expert models through a unified approach, enhancing CLIP's performance via multi-source knowledge distillation. TransAgent collaborates with 11 heterogeneous agents to empower vision-language models without adding inference phase costs. The framework achieves good performance on some visual recognition datasets.

Strengths

1. The problem is of practical importance. 2. The experiments are sufficient. 3. The gains of knowledge distillation seem strong.

Weaknesses

1. In my opinion, the proposed method leverages multiple pretty strong 'experts' for knowledge distillation, e.g., SAM, MAE, DINO, ViTDet, GPT, and Vicuna. However, almost all the baselines to be compared do not rely on external models. Hence, I think the majority of comparisons (e.g., Table 1 and Figure 4) may be unfair. The proper baselines should be other distillation methods (using the same external models as this paper), which the proposed method is actually orthogonal to most of the current baselines. 2. On top of 1, I think that if some significant technical contributions regarding knowledge distillation are not proposed, simply 'applying knowledge distillation methods to CLIP' may not be an acceptable novel contribution for me. 3. Given that multiple pretty strong 'experts' (e.g., SAM, MAE, DINO, ViTDet, GPT, and Vicuna) for knowledge distillation have been employed, the current gains of performance seem limited.

Questions

Please refer to 'Weaknesses'.

Rating

3

Confidence

4

Soundness

3

Presentation

2

Contribution

2

Limitations

The authors have addressed the limitations and potential negative societal impacts of their work.

Summary

This paper aims to handle heterogeneous foundation model combination across different pretrained backbones. Instead of using vanilla ensemble, it proposes to use a distillation process to transfer knowledges from different agents. More specifically, it uses a learnable gate module to integrate different knowledge sources. Extensive experiments show the propose method outperforms other related models.

Strengths

1. Efficiently combining several pretrained foundation models is a valuable research direction. 2. Overall, the writing is clear to read and follow, such as motivation, methodology, and empirical results. 3. Compared with baseline, the proposed model does not need to involve extra inference cost, benefitted by a distillation process. 4. Empirical results show the model superiority.

Weaknesses

1. Overall, I think this is more like a jointly distillation technique, instead of so-called agent collaboration, which may lead to certain misunderstanding. 2. Some figures are a little confused to understand. A clean and informative figures to show the whole working pipeline will be helpful. 3. Even if the empirical results are good, the proposed method still lacks of research novelty.

Questions

See the weaknesses section above.

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

None

Summary

This paper introduces a TransAgent framework, which guides CLIP to generalize with multi-source knowledge distillation. The framework contains three kinds of collaboration, including vision models, language models and muti-modal models. A Mixture-of-Agents (MoA) gating mechanism is proposed to adaptively integrate the knowledge. SOTA is achieved on 11 datasets under the few-shot scenarios.

Strengths

This method guarantees the deployment efficiency, as knowledge from heterogeneous agents are distillated and injected into CLIP. This framework can be extended flexibly. Any expert can be introduced as a teacher model. The paper is easy to follow. Figures and experiments are clearly presented.

Weaknesses

The method applies multi-teacher distillation in clip with prompts, and popular large models are exploited as teachers. Thus, the novelty is limited. Given that the framework comprises three components — VAC, LAC, and MAC—the reviewer insists that the results of every module (only-VAC, only-LAC, only-MAC) will be more persuasive to demonstrate their respective effectiveness (i.e. only-VAC's, only-LAC's, only-MAC's). The clip absorbs various knowledge from 11 excellent agents. In addition to few-shot learning on 11 downstream datasets, experiments on general dataset are also encouraged, like WebQA and CIRR. In the related works, research on muti-teacher distillation should be included. Baselines in the experiments should be introduced in the related work. Typos in line 259: knowledege -> knowledge

Questions

11 agents contain different domain knowledges. Irrelevant information or inconsistent information may be induced. Is there any challenge in the training and how to deal with challenges? How to prevent overfitting when distilled from 11 pretrained models? The λ_2 is far greater than other λ values, does it mean that the loss of LAC is more important than other losses?

Rating

5

Confidence

4

Soundness

4

Presentation

3

Contribution

3

Limitations

The authors have listed the limitation in the paper. It is encouraged to provide more analysis and solutions.