GlocalCLIP: Object-agnostic Global-Local Prompt Learning for Zero-shot Anomaly Detection

> ### Q1. Difference between AnomalyCLIP and Ours. Thank you for your question regarding the differences between AnomalyCLIP [4] and our approach. As described in General response 1, we acknowledge and appreciate the foundational contributions of AnomalyCLIP and related works. To summarize, our method, GlocalCLIP, extends AnomalyCLIP by explicitly separating global and local information and employing Glocal Contrastive Learning. This design allows for complementary learning of both global and local perspectives, which plays a critical role in improving decision-making for anomaly detection tasks. The effectiveness of our approach is demonstrated through the results presented in Fig. 3 and Tables 1 and 2. We hope this addresses your concern, and we are happy to provide further clarification if needed.

> ### Q3. Clarification of Illustrations and Rationale for Attention Mechanisms. Thank you for your question regarding anchor prompts. As explained in General response 3, the global prompt was selected as the anchor based on the hierarchical characteristics of image semantics. This choice ensures better alignment of local text embeddings relative to global normality and abnormality. For a detailed explanation and supporting results, please refer to General response 3 and the ablation study summarized in the table. If further clarification is needed, we would be happy to provide additional details. V-V attention proves to be advantageous due to its ability to preserve local visual semantics while minimizing bias and disturbance in the attention computation process. Unlike Q-Q and K-K attention, which rely on the original attention map formed by Q and K, V-V attention operates independently of the original attention map. We conducted an ablation study comparing Q-Q, K-K, and V-V attention. The results demonstrated that V-V attention effectively extracts local visual features as intended. | Attention | Industrial domain | | Medical domain | | |:---:|:---:|:---:|:---:|:---:| | | **Pixel-level** | **Image-level** | **Pixel-level** | **Image-level** | | **Q-Q** | (74.3, 46.6) | (50.1, 73.2) | (79.6, 50.2) | (41.5, 49.0) | | **K-K** | (78.5, 52.6) | (50.8, 61.7) | (78.8, 46.5) | (68.6, 71.8) | | **V-V**| (**95.3, 84.0**) | (**87.2, 89.8**) | (**90.3, 75.1**) | (**94.9, 95.4**) |

> ### Q1. Difference between AnomalyCLIP and Ours. Thank you for your question regarding the differences between AnomalyCLIP [4] and our approach. As described in General response 1, we acknowledge and appreciate the foundational contributions of AnomalyCLIP and related works. To summarize, our method, GlocalCLIP, extends AnomalyCLIP by explicitly separating global and local information and employing Glocal Contrastive Learning. This design allows for complementary learning of both global and local perspectives, which plays a critical role in improving decision-making for anomaly detection tasks. The effectiveness of our approach is demonstrated through the results presented in Fig. 3 and Tables 1 and 2.

> ### Q2. Global and Local Branch Design. Thank you for your question regarding global and local prompts. As explained in General response 2, the third row of Fig. 4 demonstrates the differences between global and local prompts. Global prompts focus on capturing the overall semantics of an object, while local prompts concentrate on detecting fine-grained, localized defects. This distinction is further supported by the visualizations in Appendix Fig. 6 to Fig. 8, which highlight their complementary roles in representing normality and abnormality. We hope this provides sufficient clarification. Please let us know if further details are required.

> ### Q3. Concerns Regarding Global Loss and GCL Loss Design. The reason for not utilizing multi-stage outputs in the global loss is that the final layer's output best captures the class semantics of the entire image. In contrast, for the local loss, multi-stage outputs were used because the reception field of the ViT changes depending on the depth of the blocks. This behavior is supported by Appendix Fig. 11 in the ViT [6]. Therefore, the local loss leverages multi-stage outputs to account for these variations. Although GCL adopts a simple structure, we believe its use is well-justified. The primary goal of GCL is to maximize the interaction between Global and Local Prompts. Its effectiveness is demonstrated in Table 3 and Figure 4. While it may appear simple, it is an optimally designed loss structure that aligns with its intended purpose. [Reference] [6] Dosovitskiy, A. (2020). An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929.

> ### Q4,5,6. Weak Introduction, Lack of Convincing Motivation, Clarification of Fig. 1. We have revised the Introduction, Related Work, and Motivation sections. After further reflection on Fig. 1, we concluded that illustrating the comparison of prompt design would better highlight the contributions of our work. In the revised Introduction, we have acknowledged prior studies more comprehensively. Additionally, the Motivation section now emphasizes the difference between global and local representations in CLIP's results, which served as the foundation for our research approach. Thank you sincerely for helping us refine and improve our paper.

> ### Q7. Marginal Improvement Over AnomalyCLIP. AnomalyCLIP demonstrates SOTA performance in ZSAD. As shown in Tables 1 and 2, as well as the Appendix, the improvements across all 15 datasets are substantial and can be considered significant.

> ### Q8. Efficiency, Computational Costs, and Training Overhead. We kindly note that training efficiency is not the primary focus of this study, and we appreciate your understanding.

My concerns have not been adequately addressed. The author should provide more explicit qualitative and quantitative comparative analyses. I have also reviewed comments from other reviewers, Reviewers #5eoh and #bsxX, who share similar concerns. However, the author has not directly provided specific comparisons. Additionally, the author should highlight the revised sections in the revision. I keep my original rating.

We regret that your concerns were not fully addressed. We have revised the manuscript by incorporating the feedback and questions from other reviewers to further enhance the contributions and additional details. As you suggested, the revised submission will clearly highlight the updated sections for your review. Additionally, detailed responses to each comment have been provided in the corresponding sections, and we kindly ask for your confirmation on those. Regarding your feedback on the insufficiency of the quantitative and qualitative results, it is still unclear which specific aspects need improvement. If you could provide further clarification on the areas that fall short, it would greatly help us refine this study and contribute to the development of our future research efforts.

> ### Q1. Difference between AnomalyCLIP and Ours. Thank you for your question regarding the differences between AnomalyCLIP and our approach. As described in General response 1, we acknowledge and appreciate the foundational contributions of AnomalyCLIP and related works. To summarize, our method, GlocalCLIP, extends AnomalyCLIP by explicitly separating global and local information and employing Glocal Contrastive Learning. This design allows for complementary learning of both global and local perspectives, which plays a critical role in improving decision-making for anomaly detection tasks. The effectiveness of our approach is demonstrated through the results presented in Fig. 3 and Tables 1 and 2. We hope this addresses your concern, and we are happy to provide further clarification if needed.

> ### Q2. Validation of V-V Attention Layer Usage. Recent studies on anomaly detection using CLIP, such as AnomalyCLIP, PromptAD, and AnoVL [5], have demonstrated the effectiveness of V-V attention. To validate this, we conducted various experiments from an image perspective. When using V-V attention, methods such as introducing a linear layer for post-adaptation or employing image-to-text cross-attention proved to be less suitable. Instead, the use of QKV attention yielded better results. Thus, we concluded that V-V attention provides a simpler and more efficient approach to learning anomalous patterns, which is why we decided to retain the overall structure in our research. To further optimize the effectiveness of this structure, we focused on developing prompt learning methodologies. The ablation study on attention is as follows: | Attention | Industrial domain | | Medical domain | | |:---:|:---:|:---:|:---:|:---:| | | **Pixel-level** | **Image-level** | **Pixel-level** | **Image-level** | | **Q-Q** | (74.3, 46.6) | (50.1, 73.2) | (79.6, 50.2) | (41.5, 49.0) | | **K-K** | (78.5, 52.6) | (50.8, 61.7) | (78.8, 46.5) | (68.6, 71.8) | | **V-V**| (**95.3, 84.0**) | (**87.2, 89.8**) | (**90.3, 75.1**) | (**94.9, 95.4**) | [Reference] [5] Deng, H., Zhang, Z., Bao, J., & Li, X. (2023). Anovl: Adapting vision-language models for unified zero-shot anomaly localization. arXiv preprint arXiv:2308.15939.

> ### Q3. Incremental Experiments Starting from AnomalyCLIP. While we acknowledge prior research, we also aimed to demonstrate the necessity of reproduction in our study. In Table 3, F2 is identical to AnomalyCLIP except for the semantic prompt design, allowing for a direct comparison with Tables 1 and 2. Through F2, we aimed to highlight the effectiveness of semantic prompt design with deep-text prompts. Next, F3 demonstrates the benefits of separating global and local prompts. The case where global and local prompts are separated but semantic design is not applied can be seen in Table 4. Finally, F4 showcases the effectiveness of GCL. We would also like to share an ablation table aligned with your suggestions to further clarify our findings. | Module | Industrial domain | | Medical domain | | |:---:|:---:|:---:|:---:|:---:| | | **Pixel-level** | **Image-level** | **Pixel-level** | **Image-level** | | **AnomalyCLIP** | (94.2, 81.0) | (86.1, 88.7) | (89.1, 72.0) | (94.4, 95.0) | | **+Semantic design** | (95.0, 82.4) | (85.6, 88.3) | (90.0, 74.5) | (94.6, 95.2) | | **+Global local branch**| (95.3, **84.0**) | (86.2, 88.5) |(90.2, 74.4) | (89.8, 91.1) | | **+GCL** | (**95.3**, 83.3) | (**86.7, 89.3**) | (**90.3, 74.8**) | (**94.9, 95.4**) | We noted improvements in the Industrial domain, with Pixel-level AUROC increasing by 1.2%, AUPRO by 3.7%, Image-level AUROC by 0.7%, and AP by 1.4%. Similarly, in the Medical domain, Pixel-level AUROC increased by 1.3%, AUPRO by 3.9%, Image-level AUROC by 0.5%, and AP by 0.4%. Please let us know if further details are required. Thank you for your valuable feedback.

> ### Q4. Discrepancy Between F4 in Table 3 and GCL in Figure 4. In Table 3, the effectiveness of GCL (F4) was influenced by the characteristics of the dataset. GCL primarily focuses on learning complementary information between global and local prompts. However, cases where GCL was less effective can be explained using the MVTecAD and VisA datasets. The key difference between these two datasets lies in the composition of objects within the images. MVTecAD mainly consists of single objects, with a focus on detecting potential defects within those objects. In contrast, VisA often includes multiple instances of objects within the same class (e.g., candles, capsules, macaroni) or overlapping defects across objects (e.g., cashew, fryum, pipefryum). Such multi-object scenarios increase the complexity of anomaly detection as defects may be confined to only one of the multiple objects. In images containing multiple objects, referring to global information sometimes introduced confusion to local information, making it more effective to focus solely on local information. This might explain the performance decline observed in certain datasets when averaged. In summary, for datasets with multiple objects, independently analyzing global and local information proved to be more effective, while combining both types of information created better synergy for single-object scenarios. Nonetheless, the separation of global and local prompts consistently outperformed existing methods, and the significance of GCL varied depending on the characteristics of the dataset.

I believe my concerns have not been adequately addressed. Firstly, the improvements based on AnomalyClip are limited. Secondly, the proposed Global Local Branch seems to offer limited enhancement in the medical field and even shows a decline at the image level. Lastly, while the qualitative experimental results indicate significant improvement in anomaly localization with GCL, this is not evident in the quantitative experiments. Additionally, I do not fully agree with the author's explanation regarding single and multiple objects in response to Q4. I suggest the author use blue color to mark changes in the revised version for easier review and submit the rebuttal promptly rather than after an extension deadline. Therefore, I decide to maintain the original score.

We regret that your concerns were not fully addressed. Regarding AnomalyCLIP, We understand that it may seem limited in scope. However, We believe that the proposed prompt learning method demonstrates strong performance within the AnomalyCLIP framework and has the potential to achieve similarly positive results in other proposed methodological structures, showcasing its scalability and effectiveness. Secondly, regarding the performance reduction of F3, please refer to the response to reviewer fHMB's question. Additionally, the performance reduction of the global local branch in the medical domain is due to the fact that simply dividing branches can lead to a loss of complementary information. This is precisely why we proposed GCL to address this issue, and We would greatly appreciate it if you could take this into account. Lastly, regarding Q4, it would be incredibly helpful for us if you could provide more details about the reasons for your disagreement, as this would allow us to make more substantial improvements. For your convenience, we have highlighted the changes in the revised version of the paper in blue. Thank you once again for your detailed review and for taking the time to engage with our work. Your feedback has been invaluable in helping us improve our research, and we truly appreciate your interest and thoughtful input.

> ### Q1. Dfference between global prompts and local prompts. Thank you for your question regarding global and local prompts. As explained in General response 2, the third row of Fig. 4 demonstrates the differences between global and local prompts. Global prompts focus on capturing the overall semantics of an object, while local prompts concentrate on detecting fine-grained, localized defects. This distinction is further supported by the visualizations in Appendix Fig. 6 to Fig. 8, which highlight their complementary roles in representing normality and abnormality.

> ### Q2. Semantic Prompt Design and Its Impact on Single-Task Performance. Thank you for your question regarding Table 4. While AUROC generally increased or remained stable, the decrease in AP within the Industrial domain can be interpreted as a reduced tendency to classify instances as anomalies. This may be due to the addition of normal learnable tokens to the prompts, which could introduce confusion when a single prompt is tasked with determining anomaly status. However, this issue appears to have been addressed by separating the prompts into global and local categories, enabling learning from distinct perspectives. Similarly, the decline in pixel-level PRO in the Medical domain seems to stem from the same issue. The addition of normal learnable tokens to the prompts likely reduced the tendency to identify anomalous regions. This can be attributed to the challenge of using a single anomaly prompt to simultaneously learn both global and local patterns. By separating the prompts into global and local categories, the global prompt effectively captures semantic anomalies from a high-level perspective, while the local prompt focuses on detecting fine-grained semantic anomalies. This separation has enabled a more effective application of the approach.

> ### Q3. Explanation for F3 Performance Drop in Medical Applications. Thank you for your question regarding Table 3. F3 enhances the learning of global and local features through Global-Local Prompt Separation, allowing for more detailed representation of each. The loss function in this study places a stronger emphasis on pixel-level information. According to the dataset statistics provided in the Appendix A, the Medical domain suffers from a lack of normal data, which increases reliance on pixel-level loss during training. As a result, the simple separation of global and local prompts may not sufficiently adjust image-level features due to the absence of pixel-level information, leading to reduced performance. To address this issue, we introduced F4, Glocal Contrastive Learning, to complement the loss of global and local information and mitigate this limitation.

> ### Q4. Explanation of Anchor Prompts. Thank you for your question regarding anchor prompts. As explained in General response 3, the global prompt was selected as the anchor based on the hierarchical characteristics of image semantics. This choice ensures better alignment of local text embeddings relative to global normality and abnormality. For a detailed explanation and supporting results, please refer to General response 3 and the ablation study summarized in the table. If further clarification is needed, we would be happy to provide additional details.