Summary
The paper presents a new technique to improve OOD, as well as IID, prediction for a pre-trained Language Vision Model (LVM). The heart of the proposed method is to learn both an ID context (i.e. perceptual context) and an OOD context (i.e. spurious context) to improve the classification of both ID classes and OOD samples. The paper proposes a new loss function to combine the ID and OOD losses together and a sampling strategy to produce spurious samples.
Strengths
The strengths of the paper are in its empirical validation and the originality around combining insights from different works. The proposed technique in the paper shows strong empirical performance, including ablation testing, on the main OOD tasks and measures. The quality of this validation well supports the claims made in the article about the need to consider both ID and some kind of spurious context when doing open-world detection.
The paper also combines elements from previous work, like the idea of having perturbed examples from VOS and NPOS, and the idea of learning prompts in the text vector space from Learning to Prompt into one framework.
Weaknesses
Despite its strong empirical validation, the paper does have some weaknesses in its clarity and novelty. Beginning with novelty, the proposed technique of CATEX seems to be only an incremental improvement on NPOS (e.g., creating perturbed examples as part of training for OOD) and directly uses the technique from Learning to Prompt with only a change in the loss function. In essence, the paper doesn’t present any insight that the NPOS/VOS papers already presented, namely the inclusion of perturbed samples into learning for an LVM can help with OOD performance. If the paper were more explicit, especially in the methods section and discussion section, about how the proposed method differs from previous ones, it would help for establishing the novelty. For example, I believe both VOS and NPOS train the underlying CLIP model, while the proposed technique of CATEX uses the Learning to Prompt technique of training a lightweight layer on top of the CLIP model. Such a change seems to balance between being good at ID tasks, while not distorting the feature space.
In terms of clarity, there is not enough detail in the methods section to both deal with the aforementioned novelty issues as well as to fully understand the training process and the perturbation guidance. For the perturbation guidance, lines 174-176 make it sound like its changing the actual words or tokens (as is done in the Kwon et al. article with masked language modeling), versus the embeddings of the text, as is done in Learning to Prompt article. If the proposed technique is actually masking the tokens, versus changing the context vectors in the embed space, then how is the training done to optimize the text, as the method in Learning to Prompt only deals with a vector space? Also, does perturbation by masking tokens fully make sense? For example, is the perturbation of “a photo of a dog” to “a [MASK] of a dog” or “a photo of a [MASK]” really a meaningful perturbation for the spurious context? I wish there was something like a walkthrough example of the perturbation as well as some more explanation of the training method and intuition behind the perturbation to better understand the contribution of the work.
Finally, there are a couple of areas where the writing could be improved. There are some sentences throughout that need to be proofread for grammar. For example, the last sentence of the abstract is a run-on sentence and the sentence on lines 48-50 is unclear in what its trying to say.
-------- Following author's responses --------------
I believe the authors have significantly addressed my concerns about the perturbation guidance. They have both added additional explanations as well as done some additional experimentation around ideas like how many tokens to mask.
Questions
1. How does the proposed method find optimal perturbations of the textual input? How does the method decide on which tokens or words to mask?
2. How could the proposed method be used when you have no labeled data (i.e. a true zero-shot setting) to improve performance? Or can it? For example, can perturbations be included in at inference time – combined with the OOD scoring function – to do zero-shot labeling?
3. What is the performance of the proposed method versus CoCoOP? While the paper does investigate the performance of its proposed method versus CoOp, and rightly concludes its does better with OOD, it does not evaluate the performance of its proposed method against the newer version of the CoOP method (i.e. CoCoOP) which was explicitly designed to deal with OOD issues of CoOP.
-------- Following author's responses --------------
I believe the authors have answered all three of my questions. In particular, I found their answer around 2, of how to use the proposed method to improve the zero-shot performance of LVMS, to be very interesting and notable. I also find it quite interesting that there was such a performance gap between the proposed method and CoCoOP.
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Limitations
The authors have addressed nearly all of their limitations and addressed those dealing with societal impact. The only limitation they have not addressed is that the proposed method still requires labeled (or captioned) data in order to work and cannot work in a zero-shot setting as was the promise of CLIP. I welcome the author’s reply on this, as I am not sure if the proposed method couldn’t be used without labeled data.