Federated Learning from Vision-Language Foundation Models: Theoretical Analysis and Method

Integrating pretrained vision-language foundation models like CLIP into federated learning has attracted significant attention for enhancing generalization across diverse tasks. Typically, federated learning of vision-language models employs prompt learning to reduce communication and computational costs, i.e., prompt-based federated learning. However, there is limited theoretical analysis to understand the performance of prompt-based federated learning. In this work, we construct a theoretical analysis framework for prompt-based federated learning via feature learning theory. Specifically, we monitor the evolution of signal learning and noise memorization in prompt-based federated learning, demonstrating that performance can be assessed by the ratio of task-relevant to task-irrelevant coefficients. Furthermore, we draw an analogy between income and risk in portfolio optimization and the task-relevant and task-irrelevant terms in feature learning. Leveraging inspiration from portfolio optimization that combining two independent assets will maintain the income while reducing the risk, we introduce two prompts: global prompt and local prompt to construct a prompt portfolio to balance the generalization and personalization. Consequently, we showed the performance advantage of the prompt portfolio and derived the optimal mixing coefficient. These theoretical claims have been further supported by empirical experiments.

Paper

Similar papers

Peer review

Reviewer YEHg5/10 · confidence 4/52024-06-23

Summary

In this papaer, the authors proposed a method to finetune vision-language model (CLIP) via prompt learning. Specifically, the author optimize a global textual prompt which will be optimized via FedAvg, while each client additionally optimize a local textual prompt. The proposed method is a combination of the existing methods CoOp and PromptFL and outperforms both methods.

Strengths

1. The paper is well-written. 2. The theoretical analysis of the proposed method is thorough and clear.

Weaknesses

1. The proposed method is basically a combination of both CoOp and PromptFL, or to be more specific, optimizing both client-specific and client-agnostic textual prompt. The novelty of the proposed needs to be further highlighted in the main paper. 2. The combination strategy seems to be a bit straight forward, i.e., using a single scalar value $\theta$, the author should discuss other possibilities. 3. The authors should compare with other prompt-based tuning methods for FL, e.g., [1][2]. Also, adding more ablation studies could be beneficial. [1] Guo T, Guo S, Wang J. Pfedprompt: Learning personalized prompt for vision-language models in federated learning[C]//Proceedings of the ACM Web Conference 2023. 2023: 1364-1374. [2] Qiu C, Li X, Mummadi C K, et al. Text-driven Prompt Generation for Vision-Language Models in Federated Learning[J]. arXiv preprint arXiv:2310.06123, 2023.

Questions

Please see the questions listed in the weakneses section.

Rating

5

Confidence

4

Soundness

2

Presentation

3

Contribution

2

Limitations

The authors do not discuss the limitation of the method.

Reviewer kpWj6/10 · confidence 2/52024-07-12

Summary

This paper discusses the integration of pretrained vision-language foundation models, such as CLIP, into federated learning. The idea is to use prompt-based federated learning to minimize the communication and computational costs. The authors go into the theoretical analysis to understand the performance of this approach

Strengths

- The authors provide a theoretical understanding on prompt Federated Learning for models like CLIP. - The results demonstrate that the combination of global and local prompts can support increased accuracies. - The overall investigation of this space is interesting and useful for the community

Weaknesses

- While the overall approach is quite interesting, the authors restrict their investigation in a much simpler classification task. It is unclear why a larger foundation model is required for simpler classification tasks, where traditional FL techniques on vanilla models (e.g., reset, mobileNet etc) might work better towards learning new unseen distributions that a foundation model might not have seen (e.g,. x-ray hospital images etc). - Following the previous comment, it is unclear how this method would work on unseen distributions that most FL applications are mostly useful for. Most FL models are quite good at generalising to data that can be found on the public internet, but it is unknown how well they would work on out-of-distribution private data. - Given the fact that there are a few related prompt-based FL methods (e.g., promtFL, FedPrompt, etc) limit a bit the novelty of this work to mostly providing a solid theoretical analysis. - There might be some privacy implications from this work, as sensitive data are shared. - The evaluation was done with mostly 10 (up to 50) users, whereas in typical FL scenarios we want to learn from million of users. It is unclear how these results would generalise. Overall, the evaluation is a bit simplistic, showing only accuracy results on a few tasks.

Questions

See above

Rating

6

Confidence

2

Soundness

3

Presentation

2

Contribution

3

Limitations

See above

Reviewer XM5m4/10 · confidence 5/52024-07-12

Summary

This paper studies theoretical properties of prompt based federated learning methods for visual language models. Following the proposed theoretical results, a new algorithm named Global-local Prompt Portfolio for Federated Learning (PromptFolio) was proposed. The proposed algorithm was examined in image classification tasks using a CLIP model.

Strengths

Analyzing prompt-based federated learning methods with feature representations and exploiting their relation to portfolio optimization is a nice idea. The proposed analyses also led to a new algorithm for prompt based federated learning.

Weaknesses

Experimental analyses are limited and should be improved as well.

Questions

In the algorithm, each client sends prompts to a server. How do assure privacy protection in this case? In the analyses, CLIP was utilized as a foundation model. However, the main claim of the paper is providing a generic framework for a more general category of visual language foundation models. Therefore, the proposed method should be examined using the other visual language models as well. Moreover, CLIP was used for image classification tasks, with limited prompts. In order to show the generalization of the proposed method for prompts with different structures, additional results should be given for the other tasks, such as image captioning and scene description, as well. In Fig. 2.b, the best accuracy is achieved for 20 users, and the lowest accuracy is achieved for 5 users. Could you please elaborate this result? Could you please provide additional analyses for a larger number of users as well?

Rating

4

Confidence

5

Soundness

2

Presentation

2

Contribution

2

Limitations

Limitations were partially addressed.

Authorsrebuttal2024-08-14

Dear Reviewer XM5m and AC, Hope this message finds you well. As the discussion deadline approaches, we sincerely hope to receive a response to our rebuttal. We believe we have properly addressed the concerns by Reviewer XM5m as our reply to these similar concerns have been recoginized by other reviewers. We observed that the concerns by Reviewer XM5m regarding privacy issues in federated prompt learning and experiments involving a larger number of users are similar to those raised by Reviewer kpWj. They have acknowledged our response and maintained their score at 6. In our response, we explained that prompts are small vectors (16k parameters) without explcit semantic meaning passed through text encoder, which **actually helps protect privacy** compared to standard federated learning setting. Besides, we illustrated the phenomenon mentioned in Question 4, where "20 users achieve the highest accuracy," by discussing **the partition strategy used for few-shot learning**. We also included experiments with a larger number of users, as shown in **Figure R3** of the PDF. The other concern by Reviewer XM5m about generalization to different tasks was also addressed for Reviewer BdA2, who **appreciated our response and increased their score from 4 to 5**. We highlighted that the classification tasks exemplify our theoretical framework and that previous work considers only classification tasks as well. Therefore, **extending to different tasks is beyond the scope of this paper**. Additionally, we demonstrated generalization to different structures through further experiments, as shown in **Figure R2** of the PDF. Thank you again for comments. We sincerely hope you can reconsider the score for our work. Best regards, Submission 6698 Authors

Reviewer bKME5/10 · confidence 3/52024-07-13

Summary

The proposed methodology offers a novel take on analyzing federated learning using prompt learning (for foundation models) via feature learning theory. At its core, the idea is to identify and monitor task-relevant and task-irrelevant features, and leveraging inspiration from portfolio optimization which says to combine independent assets to maintain income while decreasing risk of investment, construct a 2 part prompt. One part which is global and another which is local, to help with personalization. By mixing these prompts optimally, the proposed PromptFolio method highlights an improvement in overall performance of the FL system.

Strengths

- A key strength of the proposed method is its novel take on explaining prompt-based federated learning, using feature learning theory. - Clarity in idea and its link to existing work.

Weaknesses

Abstract/Conceptual Questions: - Could the authors comment about the few-shot learning aspect of foundation models (+prompt learning) and how the optimal mixing coefficient would be affected in that scenario? - An adjacent sub-field that contains foundation models and federated learning is the study of their impact in learning new classes using prior knowledge (base vs. new classes performance). Could you comment on how this sub-field could relate to the proposed method? Clarifications: - Could the authors clarify the exact dimensionality of the prompt used in each of the baselines as well as PromptFolio? - Could the authors mention if there are previous works that have proposed a 2-level prompt learning idea? (since Section 2.2 L 107 mentions that an exploration of such a idea and the cooperation between prompts is sparse) - In Section 4. L. 224, the authors mention that "task-relevant coefficients can be directly added". Could you clarify this statement? - Could the authors highlight the data setting used to generate Table 1? - Could the authors explicitly state the train and test conditions (settings) used within the experiments? Suggestions: - Since the proposal of PromptFL, there are have been further advances to the idea of prompt learning within the federated setting. Could the authors comment and draw comparisons against the more recent methods?

Questions

The questions have been included in the weaknesses section above.

Rating

5

Confidence

3

Soundness

2

Presentation

2

Contribution

4

Limitations

yes, the authors have addressed limitations within their choices of the experiment setting. However, I would recommend drawing more broad comparisons as to potential missing pieces in comparison to more recent work as well as the finer points from the feature learning theory that don't fully match up to the settings in the FL system.

Reviewer bKME2024-08-13

I thank the authors for their in-depth responses to my comments. As a follow up to their responses, - As a natural follow-up to the responses to Q1 under clarifications, have the authors observed the change in performance of baselines when switching to prompts of 16k params? - Could the authors provide more context for the selection of $\alpha=1$ and discuss the impact of lower $\alpha$ on the algorithm? (if possible also provide results) I am specifically interested in the lower $\alpha$ setting since it would allow for more in-depth and interesting analyses given the theoretical framework outlines in the manuscript.

Authorsrebuttal2024-08-13

Thank you for your response. We'd like to address your concerns. > Q1: As a natural follow-up to the responses to Q1 under clarifications, have the authors observed the change in performance of baselines when switching to prompts of 16k params? 1) First, we would like to emphasize that in our algorithm, although 16k parameters (8k global + 8k local) are used, only the 8k parameters corresponding to the global prompt are shared across clients in federated learning. Thus, our comunication cost is exact the same as baselines with 8k parameters. Using baselines with 16k parameters will double the communication cost, resulting in an unfair comparison. 2) To further alleivate your concerns, we have added experiments by increasing the parameters of baselines from 8k to 16k. From the following table, we can obsesrve that: 1) increasing the parameters will not always increase the performance; 2) PromptFolio still has the advantage even under this unfair comparsion benefit from the prompt portfolio mechanism. | | Cifar100 | DomainNet | Caltech10 | OxfordPets | DTD | | -------------------- | -------------------- | -------------------- | -------------------- | -------------------- | -------------------- | | CoOp (8k param) | 76.88 $\pm$ 0.07 | 91.83 $\pm$ 0.13 | 97.10 $\pm$ 0.20 | 87.85 $\pm$ 0.32 | 56.39 $\pm$ 0.48 | | CoOp (16k param) | 77.74 $\pm$ 0.21 | 91.92 $\pm$ 0.18 | 96.41 $\pm$ 0.43 | 87.68 $\pm$ 0.44 | 58.33 $\pm$ 0.41 | | PromptFL (8k param) | 78.16 $\pm$ 0.16 | 92.72 $\pm$ 0.16 | 95.51 $\pm$ 2.62 | 88.91 $\pm$ 0.72 | 70.99 $\pm$ 0.32 | | PromptFL (16k param) | 78.53 $\pm$ 0.33 | 92.88 $\pm$ 0.29 | 96.23 $\pm$ 1.41 | 91.81 $\pm$ 0.64 | **71.73 $\pm$ 0.31** | | PromptFolio (8k + 8k param) | **80.17 $\pm$ 0.05** | **93.04 $\pm$ 0.09** | **97.24 $\pm$ 0.11** | **92.17 $\pm$ 0.32** | 71.32 $\pm$ 0.49 | > Q2: Could the authors provide more context for the selection of $\alpha = 1$ and discuss the impact of lower $\alpha$ on the algorithm? (if possible also provide results) I am specifically interested in the lower $\alpha$ setting since it would allow for more in-depth and interesting analyses given the theoretical framework outlines in the manuscript. We conducted experiments with multiple different $\alpha$ values instead of using only $\alpha = 1$. For instance, in Table 2, we used $\alpha = 0.3$ to compare our result with baseline methods. In Figure 2(a), we conducted parameter analysis for $\alpha$ by implementing PromptFolio with different $\alpha$ values (0.01, 0.1, 1, 10). Here, we observe that a lower $\alpha$ indicates a more non-i.i.d. data distribution, requiring a larger mixing coefficient $\theta$ to retain more local information. Our theoretical analysis also supports this result. In our framework, $\Chi_k$ measures the heterogeneity across clients. A lower $\alpha$ leads to more data heterogeneity, meaning $\Chi_k$ will be lower as well. As described in Theorem 5.2 on Line 279, a lower $\Chi_k$ (lower $\alpha$) corresponds to a higher $\theta^\star$, also aligning with the experimental results in Figure 2. Further analysis on how different levels of data heterogeneity affect learning dynamics can be found in Theorem D.2.

Authorsrebuttal2024-08-14

Dear Reviewer bKME, We thanks for your comments again. As the discussion process is coming to a close, please let us know if you have any further questions. Best regards, Submission 6698 Authors

Reviewer BdA25/10 · confidence 4/52024-07-18

Summary

This paper tackles the interesting problem of analyzing federated learning (FL) from vision-language foundation models like CLIP. The authors develop a theoretical framework based on feature learning theory to understand how prompt-based FL works. They introduce a new algorithm called PromptFolio that mixes global and local prompts, drawing an analogy to portfolio optimization in finance. The theoretical analysis shows how this approach can balance generalization and personalization in FL. They back up their claims with experiments on several datasets.

Strengths

- The feature learning approach they develop seems pretty solid. - The connection between prompt mixing and financial portfolio theory is novel. - The ablation studies on data heterogeneity and client numbers are particularly informative.

Weaknesses

- This reviewer is confused by the feature learning theory (I am not familiar with it), which is introduced in a relatively short section. Is it (looks like) a common sense in deep learning or a kind of new theory proposed recently? It would be helpful to have a more thorough introduction to the theory. - Using a single activation function for the text encoder is a pretty big simplification from how CLIP and other large language models actually work. - All the experiments focus on classification tasks. Any thoughts on how PromptFolio might perform on other vision-language tasks like image captioning or visual QA in a federated setting? - Why is the proposed method on OxfordPets dataset not as good as coop? While it seems to be dominating on other datasets.

Questions

see weaknesses.

Rating

5

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

yes

Authorsrebuttal2024-08-12

Reviewer Feedback Request on Rebuttal

Dear Reviewers and AC, Thank you for your valuable comments. We have made every effort to address the concerns raised by the reviewers. We notice that none of the reviewers have provided follow-up comments on our responses. As the discussion stage deadline approaches, please let us know if you have any further questions. Thank you and best regards, Submission 6698 Authors

Reviewer YEHg2024-08-12

Thanks for the rebuttal. I have raised my score to Borderline Accept.

Reviewer kpWj2024-08-12

I would like to thank the authors for clarifying multiple points and overall providing a detailed rebuttal to all authors. After reading all reviews and comments, I think I will stay with my weak accept recommendation.

Reviewer BdA22024-08-14

Thanks for the rebuttal. Some of my concerns have been addressed. I would raise my score.

Program Chairsdecision2024-09-25

Decision

Accept (poster)

© 2026 NYSGPT2525 LLC