Response to Reviewer aTSX
We appreciate the encouraging comments that the problem and tree-probe approach are interesting and regarding the diverse experimental validation. Thank you for the opportunity to address your concerns.
**Experimental settings**: We agree to clarify and add detail in the main paper (Sec 4.1, 4.2) and supplemental (S-3) about experimental settings. We also clarify below and in our response to Reviewer ShHL.
- Our experimental settings are designed to represent different practical requirements of continual learning. In each case, labeled training examples are received in batches.
- Data incremental (Fig 2a): the batches are uniformly randomly sampled, reflecting a case when data is received, e.g. from users, and labeled.
- Class incremental (Fig 2b): each batch contains all samples of a particular category, reflecting a case where the developer intentionally finds and labels examples of new classes to increase capability.
- For data- and class-incremental, experiments are done individually for each task, and accuracy is averaged across tasks. The candidate label set for a task is constant, so accuracy is measured over all classes, even for classes that do not yet have training data.
- Task incremental (Fig 2c): A single system is trained for each task sequentially, and test accuracy is averaged across all tasks at each stage.
- Flexible inference (Fig 2d): In the task incremental setting, we also measure performance when the candidate label set does not match provided examples (zero-shot, averaging over three held-out tasks), or is the union of all tasks’ candidate labels and labels from zero-shot tasks (union + zero-shot), or is a subset of all candidate labels and labels from zero-shot tasks (mix+zero-shot). These are highly challenging scenarios that require retaining the ability to predict over arbitrary label sets, as well as the label sets that correspond to training examples.
**Paper organization**: We will consider moving the problem definition before Sec 3, as suggested.
**References**: Thank you for these references, which we will incorporate. Although [1,2] have similar goals, they use attribute-based representations, while we build on language-image representations. As mentioned to Reviewer ShHL, reported results in those papers and the Radford et al. paper show that CLIP achieves much higher zero-shot accuracy than these methods reach even after training. We confirmed this in early testing as well. This is also posted to Reviewer ShHL: In our own test, under the generalized zero-shot learning setting which requires mapping learned representations to a union of seen and unseen labels, CLIP obtains the seen and unseen accuracy of 91.7 and 94.7 on the AwA2 dataset, respectively, largely surpassing the results (seen: 60.2, unseen: 77.1) obtained by [B] after training.
**Discussion to prompt-based methods**: We agree that prompt-based methods are an important direction to explore in future work. Currently, such methods are suited mainly for task-incremental due to the large computation required. Generally, our AIM approach to combine systems can apply to any specialized model, so larger and more tuned models such as the prompt-based models will predictably improve accuracy (at cost to slower training). Table 2 shows that using a large CLIP model, for instance, increases target task accuracy from 80.5 to 90.2.