Author response (part I)
Thanks for your feedback. Here we explain your additional questions about our theoretical part as below (we reorganize the order of questions and answers for clarity).
**Q2-1: With current manuscript, I don't see a significant theoretical contribution of this paper. The outcomes seem to be a direct extension of those presented in [1] with one additional consideration, max[CIL,TAP]. Moreover, I have noticed that the notations, necessary and sufficient conditions, theorem 1, and proofs are very similar to [1].**
A2-1: Our main contribution in the theoretical part is to demonstrate the importance of WTP, TII and **TAP** for continual learning with **pre-training**. This is distinguished from the main conclusion in [1] due to the introduction of pre-trained knowledge, with the necessity of all three components empirically justified in Table 2. As the additional consideration, TAP is performed specifically on the **instructed representation space**, which is highly dependent on the pre-training backbone (i.e., $\theta$), aiming to improve predictions by coupling it with TII and WTP. Since [1] is a pioneering theoretical study decomposing the objective of continual learning from scratch, we build on its notation and analysis for the shared part (i.e., WTP and TII), in order to provide an explicit comparison and demonstrate clearly our own contribution (i.e., TAP in response to the impact of pre-training). In fact, the differences and connections of [1] and ours have been explicitly discussed in our paper (see line 231-234).
As summarized in the contribution part of the paper (line 50-56), our theoretical results provide strong insights to analyze the sub-optimality of representative methods for prompt-based continual learning and to address the sub-optimality with an innovative approach. Therefore, we respectfully argue that our theoretical contribution is not in isolation but highly synergistic with the empirical findings and performance improvements, as acknowledged by Reviewer pdqM, 6q9Z and tgkY. These results can potentially provide a unified perspective for parameter efficient fine-tuning in continual learning with pre-training, as shown in the response to your Q5 in the Rebuttal page.
**Q2-2: Regarding the author's comment A3, what is the difference between the TAP probability and CIL probability? Why TAP = TII * WTP in [1] becomes TAP != TII * WTP in this pre-training setting?**
A2-2: In this paper, the TAP probability $ P(\boldsymbol{x} \in \mathcal{X}^{y}|\mathcal{D},\theta)$ is a categorical distribution over all observed classes $\bigcup_{k=1}^{t} \mathcal{Y}k$. The typical CIL probability $P(\boldsymbol{x} \in \mathcal{X}{\bar{i},\bar{j}}|\mathcal{D},\theta)$ can be further divided into TII * WTP where $\theta$ carries the pre-trained knowledge. Without considering pre-training as in [1], TAP = TII * WTP, i.e., $ P(\boldsymbol{x} \in \mathcal{X}^{y}|\mathcal{D})=P(\boldsymbol{x} \in \mathcal{X}{\bar{i},\bar{j}}|\mathcal{D})$, due to classifying the same input in the same semantic space, i.e., $\frac{\exp(h{\psi}(f(\boldsymbol{x}))[y])}{\sum_{j=1}^{t}\sum_{y' \in \mathcal{Y}j} \exp(h{\psi}(f(\boldsymbol{x}))[y'])}=\frac{\exp(\hat{h}{\omega}(f(\boldsymbol{x})[\bar{i}])}{\sum_{i=1}^{t} \exp(\hat{h}{\omega}(f(\boldsymbol{x})[i])}*\frac{\exp(h{\psi}(f(\boldsymbol{x}))[\bar{j}])}{\sum_{j \in \mathcal{Y}i} \exp(h_{\psi}(f(\boldsymbol{x}))[j])}$.
In contrast, with considering pre-training the network $f$ with parameter set $\theta$ in our paper, TAP != TII * WTP, i.e., $ P(\boldsymbol{x} \in \mathcal{X}^{y}|\mathcal{D},\theta)\neq P(\boldsymbol{x} \in \mathcal{X}{\bar{i},\bar{j}}|\mathcal{D},\theta)$. This is because, TII * WTP is to calculate $\frac{\exp(\hat{h}{\omega}(f(\boldsymbol{x})[\bar{i}])}{\sum_{i=1}^{t} \exp(\hat{h}{\omega}(f(\boldsymbol{x})[i])}$ $*\frac{\exp(h{\psi}(f(\boldsymbol{x}; \boldsymbol{p}{i}))[\bar{j}])}{\sum_{j \in \mathcal{Y}i} \exp(h{\psi}(f(\boldsymbol{x}; \boldsymbol{p}{i}))[j])}$, while TAP is to calculate $\frac{\exp(h{\psi}(f(\boldsymbol{x}; \boldsymbol{p}{i}))[y])}{\sum_{j=1}^{t}\sum_{y' \in \mathcal{Y}j} \exp(h{\psi}(f(\boldsymbol{x}; \boldsymbol{p}{i}))[y'])}$. In other words, with prompts $\boldsymbol{p}{i}$ adapting the feature space (see Fig. 3), TAP is essentially classifying the instructed representation ($\boldsymbol{h} \in \mathcal{H}^c_{i}$ in Eq. (12)), while TII * WTP is performed on uninstructed representation ($\hat{\boldsymbol{h}} \in \hat{\mathcal{H}}_{i}^{c}$ in Eq. (11)).