Advancing Cross-domain Discriminability in Continual Learning of Vision-Language Models

Summary

The paper points out that current VLM-based incremental learning tasks face the issue of text being limited to the corresponding task. It aims to propose a method that can achieve better incremental classification performance on a broader range of texts. Specifically, the paper proposes Regression-based Analytic IncrementalLearning (RAIL), which utilizes a recursive ridge regression-based adapter to learn from a sequence of domains in a non-forgetting manner.

Strengths

1. This work introduces a Cross-domain Task-Agnostic Incremental Learning (X-TAIL) setting, which evaluates the performance on a broader range of texts about VLM's continual learning ability. 2. The paper proposes a framework RAIL, which can incrementally transfer a pre-trained VLM to multiple domains while evaluating the model's performance on both seen and unseen domains.

Weaknesses

1. Compared to existing methods, one dataset is missing. Why was CIFAR-100 removed from the forgetting benchmark? Is it because the method performs poorly on this dataset? 2. Determining whether a sample is OOD relies on learned class labels, which inevitably use label (text) information. I am concerned that this could lead to label leakage. 3. Figure 2d's graph is consistent with ZSCL, only styled differently, which I find unnecessary. It may not highlight the novelty of the X-TAIL setting.

Questions

1. Additionally, during the fine-tuning stage, is each task fine-tuned using only the text corresponding to that task to ensure no label leakage occurs during the training of each task? 2. Why is the setting for Order2 different from existing methods? I am concerned about the actual performance of this method. 3. How does Formula 1 compute one-hot with embedding, given that $Y$ and $X_e$ are of different dimensions? Could you clarify the content of the formula clearly?

Rating

5

Confidence

4

Soundness

2

Presentation

2

Contribution

2

Limitations

I am primarily concerned about the application scenarios and practical significance of the proposed setting, as well as the insufficient experimental validation.

Summary

This paper proposes a Regression-based Analytic Incremental Learning (RAIL). It utilizes a recursive ridge regression-based adapter to learn from a sequence of domains in a non-forgetting manner and decouple the cross-domain correlations by projecting features to a higher-dimensional space. Additionally, the paper introduces Cross-domain Task-Agnostic Incremental Learning (X-TAIL) setting. The paper theoretically proves RAIL’s absolute memorization on incrementally learned domains. Experiment results affirm RAIL’s state-of-the-art performance in both X-TAIL and existing Multi-domain Task-Incremental Learning settings.

Strengths

1. The proposed RAIL adopts traditional machine learning techniques (e.g., Primal and Dual forms) for dealing with continual learning is novel. 2. The idea of RAIL’s absolute memorization based on analytic techniques is very appealing. Upon a good pre-trained network, the forgetting problem no longer exists, and this is very rare in the continual learning community. 3. The paper is easy to follow overall. 4. It was difficult to handle CIL problems in multi-modal CIL. This paper extends the continual learning scenario from task incremental to class incremental without task ID. This is a good contribution. 5. Experiments and settings are overall well formulated.

Weaknesses

1. Could you explain the main difference between primal and dual forms in the CIL problem? It could be quite diffcult in this community to understand. 2. Experiemnts seems a little bit thin in the manucript. Perhaps try to move some of the experiments in the appendix to the main content.

Questions

Please refer to the Weakness.

Rating

7

Confidence

4

Soundness

3

Presentation

4

Contribution

4

Limitations

NA

Summary

Continual Learning (CL) with Vision-Language Models (VLMs) has a challenge that the model must not forget both previously learned knowledge and VLM pre-trained knowledge. Existing methods realize this by using large-scale reference data or domain identity hints, which is not practical. This paper proposed RAIL, which uses a recursive ridge-regression-based adapter to address the previously seen knowledge while using the zero-shot ability of VLMs for unseen classes. In addition, This paper proposes a novel task X-TAIL that evaluates the model on seen and unseen domains without any domain-identifier hints. The proposed method is empirically evaluated on MTIL and X-TAIL and shows state-of-the-art performance.

Strengths

+ This paper tackled a novel and practical problem, X-TAIL. It is certainly unrealistic to assume that we always have domain information, and it is more natural to consider all classes. + The proposed method seemed to have some technical novelty. It is worth commending that the proposed method is not merely an application of adapter methods such as CLIP-Adapter and Tip-Adapter, but has a recursive structure that makes it the method with affinity for CL. + The proposed method showed higher performance on both MTIL and X-TAIL than existing methods.

Weaknesses

+ I feel some parts of the paper were difficult to understand. + The explanation of the proposed method in the introduction was so unclear that I could not understand the method at all from the introduction section. The author should modify the manuscripts to prepare some belief figures to support the reader’s understanding. + I couldn’t understand how to identify ID and OOD classes in the proposed method until I read the appendix. The main manuscript should be self-contained as we can understand it without reading the appendix. + The proposed method was evaluated on the few-shot setting in the MTIL evaluation, but it was not evaluated on the normal MTIL. + The proposed method should be evaluated on the full data setting following the previous works[9, 14]. + Since the proposed method is based on ridge regression, there is concern about computational complexity when the amount of data increases. It is necessary to compare and discuss with existing methods, taking into account not only the recognition performance but also the computational efficiency. + Typos: + eq(1) ||Y-X_e||^2_F -> ||Y-X_eW||^2_F

Questions

+ Why the average Last score of Dual-RAIL is higher than that of fine-tune? Are fine-tune results not the performance in the so-called ideal situation? + How are the existing methods evaluated? How were the methods before the advent of CLIP (e.g. LwF, iCaRL) applied for CLIP? How were the MTIL methods applied for X-TAIL?

Rating

6

Confidence

3

Soundness

2

Presentation

2

Contribution

3

Limitations

+ As I mentioned in Weaknesses, the proposed method based on ridge regression requires high computational complexity, so it cannot be applicable when the amount of data is very large.

Summary

This paper proposes a novel setting called Cross-domain Task-Agnostic Incremental Learning (X-TAIL), in which the model is required to incrementally learn from multiple domains and test images from both seen and unseen domains without any domain identity. Additionally, the authors introduce two Regression-based Analytic Incremental Learning (RAIL) methods (primal form and dual form) and validate the effectiveness of these methods both theoretically and experimentally.

Strengths

1. This paper is well-written and includes clear and accurate figures and equations. 2. The method begins with an introduction to primal and dual form ridge regression, analyzing whether these non-linear projections enhance the separability of CLIP features in images from different domains. This analysis motivates the design of the RAIL-adapter, making the overall approach easier to understand. 3. It is intriguing to explore the regression-based analytic IL, leveraging non-linear projection functions from both primal and dual perspectives to enhance the expressiveness of features extracted by the pre-trained CLIP.

Weaknesses

1. There is little difference in performance between the primal and dual ridge regression methods on most datasets, as shown in Figure 6. Could you provide more analysis on this? 2. Could you compare the parameters of primal and dual ridge regression methods? What are the advantages and disadvantages of each? 3. Please provide more explanation of Figure 2. For example, what do the different colored blocks represent?

Questions

See Weakness

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

Yes