Most incremental learners excessively prioritize coarse classes of objects while neglecting various kinds of states (e.g. color and material) attached to the objects. As a result, they are limited in the ability to reason fine-grained compositionality of state-object pairs. To remedy this limitation, we propose a novel task called Compositional Incremental Learning (composition-IL), enabling the model to recognize state-object compositions as a whole in an incremental learning fashion. Since the lack of suitable benchmarks, we re-organize two existing datasets and make them tailored for composition-IL. Then, we propose a prompt-based Composition Incremental Learner (CompILer), to overcome the ambiguous composition boundary problem which challenges composition-IL largely. Specifically, we exploit multi-pool prompt learning, which is regularized by inter-pool prompt discrepancy and intra-pool prompt diversity. Besides, we devise object-injected state prompting by using object prompts to guide the selection of state prompts. Furthermore, we fuse the selected prompts by a generalized-mean strategy, to eliminate irrelevant information learned in the prompts. Extensive experiments on two datasets exhibit state-of-the-art performance achieved by CompILer.
Paper
Similar papers
Peer review
Summary
This paper presents a novel setting of incremental learning, named Compositional Incremental Learning (composition-IL). This setting differs from existing ones as it involves recognizing not only new objects (e.g., a shirt), but also their states (e.g., red) and the resulting compositions (e.g., a red shirt). To address this novel setting, the authors present a prompting-based approach that leverages a pre-trained transformer network to learn the tasks incrementally, thus sharing some similarities with existing methods such as L2P, Dual-Prompt, and CODA-Prompt. Additionally, the authors devise a three-way retrieval mechanism, devoting tailored prompt pools to state, object, and composition-level representation learning. In the experimental section, the authors assess several aspects: the comparison with the state of the art, the benefits of the three-way prompting strategy, the impact of injecting object-level information into the state-level prompt, and the contributions of each involved loss function.
Strengths
1) The paper is clear and fluent. Both the dataset and the model are clearly described. 2) Novelty. To the best of my knowledge, the experimental setting is novel and deserving. 3) While existing prompting strategies often organize the prompt pool in a flat structure, this paper advances the field by leveraging on two concepts: multiple prompts pool related to different concepts/tasks, and possible dependencies between prompts of different concepts.
Weaknesses
**Many hyperparameters/tuning complexity (major)**. From a methodological perspective, the model incorporates several loss terms, each requiring a tailored balancing hyperparameter. Additionally, other factors such as the length of each prompt and the size of the prompt pool must be considered. In truth, the authors report the chosen configuration in their experiments in the second paragraph of Sec. 5.2. As can be observed, there are many hyperparameters, which could complicate the model’s application in real-life scenarios. In this respect, the authors should provide a simplified (yet still reliable) version of their approach, pruning some loss terms and configurations that have a negligible impact on results. Indeed, by examining the results of the ablation studies, it appears that there is room for simplifying the training stage. Statistical significance (major). Concerning Tables 3 and 4, the results achieved by the final configuration of the Compiler are sometimes very close to the simpler, ablated versions. For example, consider Table 4(a) where 'no injection' is close to 'O->S', and Table 3 where 'C+S' is close to 'C+S+O'. In these cases, it is difficult to conclude whether the benefits are due to effective technical advances or should be ascribed to statistical noise. In this respect, I strongly encourage the authors to perform multiple runs for each experiment and average the final results. This approach would render the comparisons more significant. **More rehearsal baseline (major)**. I would suggest including in the experimental section the results for DER++ [b] and ER-ACE [c], two straightforward baselines based on rehearsal. As the main contribution of this paper concerns the introduction of a new experimental setting, I believe it is important to provide an extensive evaluation of the existing literature. Moreover, the authors should compare their approach with SLCA [d], a recently introduced baseline for the continual fine-tuning of pre-trained models. Relation with existing works (major). There is a recent research line on multi-label incremental learning; see [a], one of the most cited papers in the field. In this respect, how does the proposed setting deviate from the multi-label scenario? To justify the introduction of a novel setting, the authors should discuss potential overlaps with existing benchmarks more thoroughly. From a more methodological perspective, instead, I would encourage the authors to provide more references while introducing the learning objectives used in their approach. For example, CODA-Prompt devises orthogonality constraints between prompts, sharing strong similarities with the objectives outlined by Eq. 2 and 3. Moreover, aligning the query and the selected prompts is common in prompting-based approaches (see L2P). **Lacking ablation studies (minor)**. In Tab. 3, it would be important to see the results of single-pool prompt learning, especially for ‘S’ and ‘O’ (‘C’ has instead been already provided). Indeed, I would guess that a simple strategy based only on object-level prompts would be enough to reach satisfactory performance. **Clarification requested (minor)**. In Sec. 4.2, the authors discuss an attention-based mechanism to inject object-level cues into the state-level prompt. In this respect, they use three learnable projections, indicated by W_Q, W_K, and W_V. However, as these transformations are continuously updated while tasks progress, did the authors consider that these projections could suffer from catastrophic forgetting? It seems that there are no explicit countermeasures against forgetting regarding these variables, which could hinder the efficacy of the retrieval stage. **Formatting issue (minor)**. Table 4 appears before Table 3. [a] Kim, C. D., Jeong, J., & Kim, G. (2020). Imbalanced continual learning with partitioning reservoir sampling. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XIII 16 (pp. 411-428). Springer International Publishing. [b] Buzzega, P., Boschini, M., Porrello, A., Abati, D., & Calderara, S. (2020). Dark experience for general continual learning: a strong, simple baseline. Advances in neural information processing systems, 33, 15920-15930. [c] Caccia, L., Aljundi, R., Asadi, N., Tuytelaars, T., Pineau, J., & Belilovsky, E. (2021). New insights on reducing abrupt representation change in online continual learning. arXiv preprint arXiv:2104.05025. [d] Zhang, G., Wang, L., Kang, G., Chen, L., & Wei, Y. (2023). Slca: Slow learner with classifier alignment for continual learning on a pre-trained model. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 19148-19158).
Questions
I have no questions.
Rating
6
Confidence
5
Soundness
3
Presentation
3
Contribution
2
Limitations
The limitations are briefly discussed in the supplementary materials.
We appreciate the reviewer EVGX’s willingness to engage in further discussion. The followings are our point-point answer: **Q1: About hyperparameters and complexity.** **A1:** We appreciate your suggestion very much. We thereby have acted on it by developing a simplified model to reduce the number of hyperparameters. Specifically, we focus on the 'C+S+O' configuration, namely Sim-CompILer, which eliminates the directional decoupled loss, RCE loss, and Object-injected State Prompting, accordingly removing $\lambda_1$, $\lambda_2$, and $\alpha$ in the total loss cost. As a result, Sim-CompILer is constrained by vanilla CE loss and a surrogate loss, represented as $L_{total}=L_{CE}+\lambda_3L_{sur}$, where the remaining hyperparameter is $\lambda_3$ only, which controls the balance between CE and surrogate loss. Apart that, $\beta$, is still needed to adjust a balance between compositions and primitives. The results, shown in the table below, show that, while CompILer performs best across both datasets, Sim-CompILer slightly lags behind but still achieves second-best results compared to the baselines in Table 1 and Table 2. We opted for three pools rather than two due to poor performance with dual prompt pools in Split-UT-Zappos, as noted in our previous rebuttal. We will include the results for this Sim-CompILer alternative in the final version. Hopefully, this simpler alternative would provide more flexibility for the potential users to conduct their related research. | Name | Split-Clothing (5 tasks) | | | | | Split-UT-Zappos (5 tasks) | | | | | |---------------|--------------------------|---|---|---|---|---------------------------|---|---|---|---| | | Avg Acc | FTT(↓) | State | Object | HM | Avg Acc | FTT(↓) | State | Object | HM | | Sim-CompILer | 88.22 | 8.37 | 90.95 | **96.47** | 93.36 | 46.43 | 19.31 | 56.83 | **79.58** | 66.31 | | CompILer | **88.74** | **6.98** | **91.61** | 96.34 | **93.92** | **47.06** | **18.84** | **57.52** | 79.53 | **66.75** | **Q2: About forgetting** **A2:** We apologize our explanation confused you due to the complex biological theory. We acknowledge that we have not implemented explicit methods to circumvent catastrophic forgetting, and this choice was intentional for the following reasons: First of all, following L2P, our model avoids catastrophic forgetting by freezing a ViT backbone across all incremental sessions. Thanks to this simple yet effective strategy, L2P and its improved variants (such as Dual-Prompt, CODA-Prompt, LGCL and our CompILer) achieve much lower forgetting rates than EWC and LwF, as shown in Table 1 of the submitted manuscript. In addition, following the suggestion from Reviewer x9g2, we perform more comparison with NCM and FeCam. While these two methods are not built based on L2P, they still suggest freezing the backbone to overcome the catastrophic forgetting effectively. Inspired by these results, we can note that the core of learning without forgetting relies on a frozen backbone, and a limited number of learnable parameters have less impact on raising the forgetting. This finding is consistent with the fact that L2P did not exploit any explicit countermeasures between the learnable prompts to alleviate the forgetting yet. Akin to the learnable prompts, we advocate learning the parameters in the object-injected state prompting module devoid of extra explicit countermeasures. On the other hand, we would like to emphasize again that the aim of continual learning is to pursuit an optimal balance between stability and plasticity. Since the stability has been obtained by freezing the backbone, another challenge is how to improve the plasticity when adapting the model on new tasks. Thanks to the learnable parameters in the prompt pool and object-injected state prompting module (even though their number is slighter than that of the backbone), it allows us to improve the adaptability of the model and increase the accuracy further. Last but not least, the model inclines to forget the compositions rather than objects and states, because the object and state primitives may reappear in new tasks. Hence, we apply object-injected state prompting to the primitive branches, leading to minor forgetting on the compositions.
The authors have answered my questions. I would like to thank them for their efforts, which address most of my initial concerns. As a result, I will raise my score to weak accept.
Summary
This work presents a new task called Compositional Incremental Learning (composition-IL). This new task extends the existing class incremental learning to a more fine-grained scenario for more realistic applications. This work formulates and designs a new composition-IL benchmark based on Clothing16K and UT-Zappos50K datasets. Technically, the authors propose several novel strategies based on prompting on pretrained ViT. Specifically, this work suggests using three prompt pools for states, objects, and their compositions respectively. To overcome the gap between the pretrained task and the composition task, the object prompts are further injected to guide the selection of state prompts. The authors also suggest a learnable generalized-mean prompt fusion scheme for prompt reducing. Extensive experiments are provided to evaluate the models.
Strengths
1. This paper has done a great job in presentation, with clear wiring and figuring, enabling the fast understanding of reviewers familiar with continual learning. 2. The contribution of proposing novel and more realistic tasks is always welcomed. And this work has done some essential initial jobs such as creating a benchmark for composition-IL. 3. The proposed method looks convincing to me.
Weaknesses
1. The main concern for me is the proposed composition-IL itself. From the experimental results (Tables 1 and 2), it seems that the previous prompt-based works can also handle this task with comparable performance. So my double is that if this work overclaims the difficulty of composition-IL? As someone working in the continual learning field, I think new settings and tasks are always welcomed only if the new task is challenging and practical enough. 2. The effectiveness of the proposed components. As shown in Table 3, I think the performance gap between C+S+O and C+S/C+O is small, so the effectiveness of multi-pool prompt learning is not that significant. Similar results are also in Table 4. In Table 4(a), there is only a 0.32% improvement in HM when introducing the object injection. In Table 4(b), there is also only a 0.3% improvement when using the GeM instead of the naive mean pooling.
Questions
Please see the 'Weakness' section for detailed questions. In conclusion, I expect the authors to provide their thoughts about the necessity of the proposed composition-IL setting, as well as some explanations of the small performance gap when equipped with the proposed modules.
Rating
6
Confidence
5
Soundness
3
Presentation
3
Contribution
3
Limitations
The authors have adequately addressed the limitations and potential negative societal impact of their work.
Summary
This paper introduces a novel task termed Compositional Incremental Learning (composition-IL), which aims to enable models to recognize a variety of state-object compositions incrementally. The authors propose a new model called CompILer, which employs multi-pool prompt learning, object-injected state prompting, and generalized-mean prompt fusion to overcome the challenges in this task. The study utilizes two newly tailored datasets, Split-Clothing and Split-UT-Zappos, and demonstrates state-of-the-art performance through extensive experiments.
Strengths
1. The introduction of composition-IL addresses a critical gap in incremental learning by focusing on the recognition of state-object compositions, which traditional class-IL and blur-IL approaches overlook 2. The proposed CompILer model is well-conceived, leveraging multi-pool prompt learning, object-injected state prompting, and generalized-mean prompt fusion to enhance the learning of compositions.
Weaknesses
1. The paper shows that CompILer only slightly outperforms LGCL and CODA-Prompt on the Split-Clothing and Split-UT-Zappos datasets. However, the authors do not detailedly analyze why LGCL and CODA-Prompt perform well in these tasks or why CompILer's improvements are limited. A detailed comparative analysis is needed. 2. Lack of analysis on model efficiency, especially on the size of learnt prompts comparing to current prompt-based methods 3. The paper lacks a systematic analysis of the hyperparameters $\lambda_1$ and $\lambda_2$ in Equation 9.
Questions
Model Efficiency: How does the size of the learnt prompts in CompILer compare to current prompt-based methods? Please provide a detailed analysis of model efficiency. Hyperparameter Analysis: Can you provide a systematic analysis of the hyperparameters $\lambda_1$ and $\lambda_2$ in Equation 9? How do variations in these values impact the model's performance? Performance Comparison: Why do LGCL and CODA-Prompt perform well on the Split-Clothing and Split-UT-Zappos tasks, and what factors limit the performance improvements of CompILer over these methods? A detailed comparative analysis would be beneficial.
Rating
6
Confidence
4
Soundness
3
Presentation
3
Contribution
3
Limitations
The authors have acknowledged the limitations of their work, particularly concerning the performance challenges on the Split-UT-Zappos dataset due to long-tail distributions and the difficulty in distinguishing state classes.
Summary
The paper propose compositional Incremental Learning to enable models to recognize state-object compositions incrementally. The paper provides two tailored datasets for composition-IL by modifying two existing datasets in the fashion domain. The paper propose a new prompt-based model comprising of multi-pool prompt learning, object-injected state prompting and generalized-mean prompt fusion. The method achieves competitive performance on the two proposed datasets.
Strengths
1. The paper discusses a new direction which aims to exploit state primitives of objects for incremental classification. The authors propose a new incremental setting where objects or states can reappear in new classes in new tasks. 2. The paper provides two curated datasets in the fashion domain to study composition-IL. 3. The paper propose a new prompt-based model for composition-IL. It is nice to report HM accuracy for better evaluation.
Weaknesses
I have major concerns with the experimental part of the paper. 1. Hyperparameters: It is very concerning that six hyperparameters are tuned on the testing set. The hyperparameter values even differ for different task splits on the same dataset. It is weird that the authors use 25 epochs for one dataset, while for the other dataset, 10 epochs for 5-task setting while 3 epochs for the 10 task setting. Even different learning rates are used for different settings on same datasets. How are these decided? It looks like everything is optimized for the test sets in all settings. This is not a fair way of doing experiments. It is acceptable if the authors fine-tune the model for one dataset and use the same parameters for all settings across different datasets (I think this is commonly done in continual learning domain). How can the proposed method be useful/practical if it needs to fine-tune so many hyper parameters on every setting using the test set to get good results? 2. Lack of experiments with random seeds: The experiments are conducted using a single random seed. The proposed method has improvements of 1% or even less in some settings. It is standard practice in CL to report results with multiple random seeds for fair evaluation and to establish the robustness of the model. 3. Competitive recent baselines like HiDe-prompt [1] are not included in the comparison. 4. Simple methods like NCM classifier [2] and Mahalanobis-distance based classifier [3] outperforms prompt methods like L2P on several datasets with first-task adaptation and no training in new tasks (using frozen model after the first task and doing continual evaluation). It would be interesting to see how these strong baseline methods work in the proposed settings. [1] Wang, Liyuan, et al. "Hierarchical decomposition of prompt-based continual learning: Rethinking obscured sub-optimality." Advances in Neural Information Processing Systems 36 (2023). [2] Paul Janson, et al, A simple baseline that questions the use of pretrained models in continual learning. arXiv preprint arXiv:2210.04428, 2022. [3] Dipam Goswami, et al, Fecam: Exploiting the heterogeneity of class distributions in exemplar-free continual learning. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
Questions
1. I am curious what is the difference between object-state pairs and a group of concepts describing a class? Like why limit the concepts to just object and state, there can be more concepts attached to a class. So, a class can also be described as a group of concepts and classes can then be learned incrementally with overlapping concepts from old classes.
Rating
6
Confidence
4
Soundness
3
Presentation
3
Contribution
3
Limitations
Limitations of this work is not explicitly addressed in paper.
Thank you very much for further discussion and comments. We appreciate very much for your raising the rating. We are happy to answer your questions below: **Q1: About hyperparameters.** **A1:** We have acted on your suggestion to reduce the number of hyperparameters by streamlining the model. Specifically, we implement a simplified alternative model (namely Sim-CompILer) , which removes the directional decouple loss, RCE loss, and Object-injected State Prompting from the original CompILer, thereby eliminating hyperparameters $\lambda_1$, $\lambda_2$, and $\alpha$, Note that, $\lambda_3$ still remains in order to balance CE loss and surrogate loss, and $\beta$ is still needed to adjust the trade-off between compositions and primitives. The results, shown in the table below, demonstrate that, even though CompILer achieves the best performance across both datasets, Sim-CompILer slightly underperforms but still ranks second. We agree with your suggestion and believe this simplified version is more practical and easy-to-follow. We will include the results for Sim-CompILer in the final version. Hopefully, this simpler alternative would provide more flexibility in potential practical applications. | Name | Split-Clothing (5 tasks) | | | | | Split-UT-Zappos (5 tasks) | | | | | |---------------|--------------------------|---|---|---|---|---------------------------|---|---|---|---| | | Avg Acc | FTT(↓) | State | Object | HM | Avg Acc | FTT(↓) | State | Object | HM | | Sim-CompILer | 88.22 | 8.37 | 90.95 | **96.47** | 93.36 | 46.43 | 19.31 | 56.83 | **79.58** | 66.31 | | CompILer | **88.74** | **6.98** | **91.61** | 96.34 | **93.92** | **47.06** | **18.84** | **57.52** | 79.53 | **66.75** | **Q2: Comparison with baselines.** **A2:** Thank you for pointing out this issue. After checking our experiments, we confirmed that there was an error in the implementation of NCM and FeCAM. Concretely speaking, FeCAM provides two official implementations, one under the Avalanche Library [1] and the other under PyCIL [2]. Since the Avalanche Library includes both NCM and FeCAM, we conducted our experiments within this framework to ensure a fair comparison. Unfortunately, during the process, we overlooked the need to update the class mask, determining which classes have been seen during training the model incrementally. This negligence leads to the unreasonable results you have noticed. Also, we made the same mistake when implementing the NCM method, which also obtained unsatisfactory results. To correct the error, we have fixed the mask and re-conducted the experiments from scratch. The new results, as shown in the following table, verify the fact that FeCAM should achieve higher accuracy than NCM. CompILer outperforms NCM and FeCAM by enhancing adaptation through the tuning of learnable prompts across all sessions. In contrast, NCM and FeCAM are limited by their inability to update the model during incremental learning. Additionally, it is noteworthy that CompILer significantly surpasses NCM and FeCAM in Object accuracy. This is because CompILer freezes the backbone across all sessions and leverages the pre-trained parameters, while the other two methods update the backbone in the first session, which weakens their effectiveness in object recognition. We will include these methods as baselines in the next version and provide proper citations. | Method | Prototype | Avg Acc | State | Object | HM | |----------|-----------|---------|--------|--------|--------| | NCM | ✓ | 31.09 | 41.91 | 39.71 | 40.78 | | FeCAM | ✓ | 33.71 | **46.32** | 40.44 | 43.18 | | CompILer | ✗ | **34.66** | 45.82 | **77.06** | **57.47** | [1] Avalanche: an End-to-End Library for Continual Learning (CVPRW 2021) [2] Pycil: a python toolbox for class-incremental learning (SCIENCE CHINA Information Sciences 2023)
I appreciate the authors efforts in improving the paper and clarifying most of my concerns. It is good that the authors propose a simpler version of the method which still works with lesser hyper-parameters and should be included in the paper. I expect the authors to have the discussion in the paper on the need to reduce the hyper-parameters and use simpler models. It is interesting to see that the baselines of NCM and FeCam performs very competitively with the prompt-based methods and is better in predicting the state while worse in object prediction. It is good to have a detailed discussion on comparison with these frozen-model baselines which highlights how the learnable prompt-parameters (also using frozen model) are affecting the performance. It is also good to add them in the model efficiency analysis (like in Table 3 rebuttal) since they do not learn any new parameters. Overall, I am now quite positive about the contribution of this work which opens up a very interesting direction in Composition-IL with very intuitive and novel setting, method, datasets and very good analysis. The extensive experiments (including the baselines and also more rehearsal-based method comparisons added during rebuttal) makes it a good benchmark for future works. One minor remark for the final version would be to add proper informative captions for all figures in the paper which now is quite empty, it makes the paper more readable and easy to understand. I raise my rating to 6.
I sincerely thank the reviewers for the efforts they spent during the rebuttal period. I still have several considerations and questions for the authors. **Hyperparameters and complexity**. While I acknowledge that tuning several loss coefficients is a standard practice in deep learning, my concerns still persist. Therefore, I urge the authors to reconsider their approach in the final version and, if feasible, reduce the number of terms involved during optimization. Based on the experimental results, there appears to be some flexibility to achieve this reduction. It's important to keep in mind that future users will need to apply this method to new benchmarks, and the abundance of hyperparameters could make this process quite challenging. **Forgetting** In my original review, I expressed some concerns regarding the learnable projections W_Q, W_K, and W_V, which are likely to suffer from catastrophic forgetting as no explicit countermeasures have been taken to protect these parameters. The authors' response, which relies on biological arguments, is somewhat difficult for me to follow. Could the authors provide a more practical discussion on this point? I believe addressing this issue is important.
I appreciate the detailed rebuttal for the authors which answers most of my concerns. 1. Hyperparameters: While the authors argue that several papers use different training hyper-parameters for different settings even using the same dataset, I still believe that this is not a robust solution, it is just optimizing the results on the test set which does not make sense. Just saying that some existing papers do the same overfitting on the test set is not enough. I would urge the authors to reduce the number of hyper-parameters (following the discussion with Reviewer EVGX) to make the method usable and more realistic. Improving the accuracy by small margins is not everything, what's more important is how robust and practical your solution is. 2. Comparison with baselines: It is appreciable that the authors reproduced the baseline methods. HiDE-prompt was expected to do better, it's good to have a discussion in the paper about the trade-offs. NCM is a baseline which can be considered as a lower bound since it does not update the model. FeCam is naturally expected to do better than NCM since it uses Mahalanobis-distance and has improved over NCM using ViTs [3]. So, it's quite surprising to see that it gets worse. I would ask the authors to look into the implementation or find appropriate justifications for this since FeCam results in Table 2 looks like an error or faulty implementation. It is important to discuss these baselines since they form the lower bounds and shows the scope of improvement using the prompt methods. I still maintain the concerns for the hyper parameters and urge the authors to look into this. I raised my rating to 5.
I thank the authors for their rebuttal. The rebuttal solved part of my concerns, so I will keep my score unchanged.
Dear Authors, Thank you for your clear responses and additional analyses. I appreciate your work and believe it adds valuable contributions to the field. I raise my rating to 6.
Decision
Accept (poster)