Visual Fourier Prompt Tuning

Summary

This paper introduces Visual Fourier Prompt Tuning (VFPT), a novel approach to parameter-efficient fine-tuning (PEFT) for large-scale Transformer-based vision models. VFPT integrates Fast Fourier Transform (FFT) with prompt tuning, allowing the model to adapt to new tasks effectively by utilizing both spatial and frequency domain information, especially when there is a significant disparity between the datasets used in pre-training and fine-tuning phases. This method not only retains the simplicity and intuitiveness of standard visual prompt tuning but also enhances model performance across various tasks without substantially increasing the parameter count. Empirical results show that VFPT outperforms several state-of-the-art PEFT methods on benchmark datasets like VTAB-1k and FGVC, achieving higher accuracy with fewer parameters. For instance, VFPT uses only 0.66% of the total model parameters while achieving 73.20% mean accuracy on VTAB-1k, surpassing both full fine-tuning and conventional visual prompt tuning methods.

Strengths

1. The paper is well-structured and clearly written, with each section logically flowing into the next. 2. VFPT introduces a unique combination of Fast Fourier Transform (FFT) with visual prompt tuning in Transformer-based models, which leverages both spatial and frequency domain information. 3. The empirical results are robust, demonstrating VFPT's superiority over existing state-of-the-art parameter-efficient fine-tuning methods. The paper includes detailed comparative analyses, showing VFPT's performance advantages across multiple benchmark datasets. 4. VFPT has significant implications for the scalability and efficiency of Transformer-based models in vision tasks. The ability to maintain high performance with fewer parameters makes large-scale models more accessible and practical for a wider range of applications. 5. The paper discusses the theoretical implications of integrating FFT within visual prompt tuning, noting improvements in the optimization landscape and model interpretability. VFPT promotes a smoother loss landscape, which correlates with better generalization and lower test error rates.

Weaknesses

1. The abstract states that the proposed method, VFPT, utilizes only 0.57% of the model parameters to achieve a performance of 73.20% mean accuracy on VTAB-1k dataset. However, Table 1 contradicts this by listing the parameter usage as 0.66%. This discrepancy raises concerns about the accuracy of the reported statistics. 2. The paper does not fully elucidate the complexity introduced by integrating FFT into the prompt tuning process. While the method is praised for its parameter efficiency, the computational overhead, especially in terms of runtime and memory consumption during the FFT operations, is not thoroughly discussed. 3. Table 3 reveals that VFPT appears to underperform relative to some established partial tuning and extra module methods when applied to models pretrained with MAE self-supervised objective. This might indicate that VFPT's reliance on Fourier transforms does not synergize as effectively with the feature representations or data distributions typical of self-supervised learning models. Understanding why VFPT performs less effectively in this context is crucial for refining the approach or identifying its optimal application scenarios.

Questions

Table 5(b) explores the effects of different Fourier Prompt Locations (Prepend, Append, and Random) on the performance of Visual Fourier Prompt Tuning (VFPT). The results suggest variations in performance based on the placement of these prompts within the input sequence. Since VPT mentions that different prompt locations are mathematically equivalent, I wonder why the location of Fourier prompts impacts model performance? The following sentence is from section 3.2 of the original VPT paper: “Notably for ViT, $x_N$ is invariant to the location of prompts since they are inserted after positional encoding, e.g., $[\mathbf{x_0}, \mathbf{P}, \mathbf{E_0}]$ and $[\mathbf{x_0}, \mathbf{E_0}, \mathbf{P}]$ are mathematically equivalent. This also applies to VPT-Deep.”

Rating

7

Confidence

5

Soundness

3

Presentation

3

Contribution

3

Limitations

The limitations have been discussed in Appendix S9.

Summary

This paper introduces Visual Fourier Prompt Tuning (VFPT), an approach for parameter-efficient fine-tuning of large vision models. VFPT integrates Fast Fourier Transform (FFT) operations into visual prompt tuning, allowing it to incorporate both spatial and frequency domain information. The method demonstrates superior performance and generalizability across various tasks and datasets, particularly when there are significant disparities between pretraining and fine-tuning data. VFPT outperforms several state-of-the-art baselines on benchmarks like VTAB-1k and FGVC, using only a small fraction of trainable parameters. The authors provide extensive empirical results, ablation studies, and visualizations to demonstrate the effectiveness of their approach. They also explore the optimization landscape and interpretability aspects of VFPT.

Strengths

Originality: The paper presents an approach by integrating Fast Fourier Transform (FFT) operations into visual prompt tuning. This is an original idea that bridges frequency domain analysis with parameter-efficient fine-tuning of large vision models. The authors draw inspiration from human visual cognition to incorporate both spatial and frequency domain information, which is a creative angle not explored in previous visual prompt tuning methods. Quality: - Comprehensive experiments are conducted across multiple benchmarks (VTAB-1k, FGVC) and model architectures (ViT, Swin Transformer). - Rigorous ablation studies examine various components of the proposed method. - The authors provide in-depth analysis of the optimization landscape and interpretability aspects. - Error bars and statistical significance are reported for main results. - The paper includes thorough comparisons with state-of-the-art baselines. Clarity: The paper is well-structured and clearly written. Significance: - It addresses a key challenge in parameter-efficient fine-tuning - performance degradation when there's a large disparity between pretraining and fine-tuning datasets. - The method achieves strong performance while using only a small fraction of trainable parameters, which is crucial for adapting large models efficiently. - The approach is general and can potentially be applied to various vision transformer architectures.

Weaknesses

1. The paper lacks a rigorous theoretical analysis of why integrating Fourier components improves generalization. While empirical results are strong, a deeper theoretical investigation could provide insights into: - Why frequency domain information helps with dataset disparities - How the balance between spatial and frequency information affects model behavior - Potential limitations or failure cases of the approach Suggestion: Develop a theoretical framework explaining the interplay between spatial and frequency domain information in prompt tuning. This could involve analyzing the properties of the loss landscape or examining how Fourier components affect the model's representation space. 2. The Fourier percentage (α) is a crucial hyperparameter, but its selection process is not fully automated. The paper suggests using dataset disparity as a guideline, but this may not always be easily quantifiable. Suggestion: Develop an adaptive method to automatically determine the optimal Fourier percentage based on task characteristics. This could involve a small meta-network that predicts the optimal α, or a dynamic adjustment mechanism during training. 3. While the paper mentions a slight decrease in training speed (2.8% on VTAB-1k), a more comprehensive analysis of computational trade-offs is missing. This is important for practitioners considering adopting the method. Suggestion: Provide a detailed analysis of computational overhead across different settings, including: - Training time comparisons with baselines - Memory usage - Inference time implications - Potential optimizations to reduce overhead 4. The paper primarily uses FFT as a black-box operation. A deeper exploration of how different frequency components contribute to performance could yield additional insights. Suggestion: Conduct experiments analyzing the impact of different frequency bands on model performance. This could involve selectively filtering certain frequency components or visualizing which frequencies are most important for different tasks. 5. The generalization to other architectures is not clear. While the method is tested on ViT and Swin Transformer, its applicability to other architecture types (e.g., convolutional networks, MLP-Mixers) is not explored. Suggestion: Extend the experimentation to a broader range of architecture types to demonstrate the method's generality. If challenges arise, analyze and discuss the limitations of applying VFPT to different model families. 6. The paper doesn't deeply explore how Fourier prompts interact with regular prompts or how they influence different layers of the network. Suggestion: Conduct a layer-wise analysis of how Fourier prompts affect feature representations throughout the network. Visualize and quantify the interactions between Fourier and regular prompts to provide insights into their complementary roles. 7. The paper doesn't thoroughly address the robustness of VFPT to different initializations or potential instabilities during training. Suggestions: Perform a comprehensive stability analysis, including: - Sensitivity to different random seeds - Learning rate schedules that work well with VFPT - Potential gradient issues or training instabilities - Robustness to different prompt lengths and configurations 8. While benchmark performance is strong, the paper lacks discussion on how VFPT performs on real-world, large-scale applications outside standard benchmarks. Suggestion: Include case studies or experiments on practical, large-scale vision tasks (e.g., autonomous driving, medical imaging) to demonstrate the method's effectiveness in real-world scenarios. Discuss any challenges or modifications needed for such applications.

Questions

Please refer to the weaknesses section.

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

Limitations: The authors do discuss limitations in Section S9 of the appendix, which is a positive point. They specifically mention the introduction of an additional hyperparameter (Fourier percentage) and acknowledge that while dataset disparity can guide its selection, an automatic search method could further improve efficiency. This shows awareness of a key limitation in their approach. However, the limitations discussion could be more comprehensive. For example: 1. They could elaborate on potential scenarios where VFPT might not perform well. 2. Discussion of scalability limitations or computational constraints for very large models is not clearly addressed. 3. Potential limitations in applying VFPT to other domains or model architectures could be explored further. Societal Impact: The authors do discuss social impact in Section S9 of the appendix, which is commendable. They highlight positive aspects such as: 1. Potential for improving model accuracy without substantial computational overhead. 2. Applicability in computationally-sensitive real-world applications. 3. Advancements towards achieving generality across datasets. However, the discussion of potential negative societal impacts is limited. The authors could have explored: 1. Potential misuse of more efficient and generalizable models in privacy-invading or surveillance applications. 2. Environmental impacts of encouraging more fine-tuning experiments. 3. Potential biases that might be introduced or amplified through this method.

Summary

This paper proposes Visual Fourier Prompt Tuning (VFPT) to address performance degradation in parameter-efficient finetuning (PEFT) methods caused by dataset disparities. VFPT integrates Fast Fourier Transform (FFT) into prompt embeddings, enhancing performance with minimal parameter usage. This work is built upon the VPT method. The key innovation is to integrate frequency domain information using FFT to VPT. The performance is evaluated on VTAB-1k and FGVC.

Strengths

The experiments are comprehensive and performance on several benchmark are impressive as well. Study of Interpretability shows some interesting findings can be helpful for future research. The ablation study is completed. We can see compared with VPT baseline, FFT’s effectiveness has been well demonstrated. The demonstration including figure/tables of this paper is very well.

Weaknesses

Overall, this is a well-organized paper. I have several questions regarding the methods and results; 1. What is the larger model’s performance when applying the VFPT? Whether the VFPT can work well with large models is an important question that hasn't been answered in the current version. As the model size grows, efficient, prompt learning can be more critical. However, current results are mainly on a relatively small size of models. 2. Whether VFPT can be integrated into the CLIP model to enhance zero-shot performance is also an interesting question. I am very interested in vision and text encoders and whether VFPT can be applied to CLIP models. 3. Current results do not answer the first question in the introduction: “Can prompt tuning generalize across datasets with varying disparities?”. It confuses me because I think of the transferable prompt design, which is trained on one dataset and then transferred to another. This motivation, to me, needs a further rephrase or validation.

Questions

I have several expectations for seeing the potential of VFPT. If the author can address these questions, I will change my scores.

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

Yes, the authors discussed the limitation of their methods.

Looking forward to the discussion

Dear Reviewers, We sincerely appreciate the time and effort you've devoted to reviewing our work. We understand that your schedule may be quite busy, especially during the weekend. As the authors-reviewer discussion phase draws to a close, we kindly request your attention to our responses (special thanks to Reviewer TrZ7 for acknowledging our response). Our aim is to gain insights into whether our responses effectively address your concerns, and to ascertain if there are any additional questions or points you would like to discuss. We look forward to the opportunity for further discussion with you. Thank you for your thoughtful consideration. Best regards,\ The Authors

Thanks for the rebuttal

The improvement over scaling up of models and CLIP is not marginal; hence, it enhances the authors' claim.