SeTAR: Out-of-Distribution Detection with Selective Low-Rank Approximation

Thanks for the rebuttal

My concerns are addressed well and I'd like to raise my score to 6. For point 2, please make it clear in the manuscript that the performance improvement is relative (by default I think people assume/expect absolute improvement being discussed). For point 3, I strongly recommend including the near-OOD results in the main text so that later works can continue to work the meaningful & challenging near-OOD detection problem (rather than keep working only on far-OOD by following previous works).

Comparisons with CLIPN and NegLabel

We apologize for the late reply and appreciate your patience. # 1. Comparisons with CLIPN - CLIPN [1] is pre-trained on the **CC-3M** dataset for **10 epochs**, involves training an additional NO-encoder for over 64 million parameters - On ViT-B/16, CLIPN-C achieves an FPR95 of 38.59 and an AUROC of 86.35, while CLIPN-A achieves an FPR95 of 31.10 and an AUROC of 93.10. - In contrast, our SeTAR method does not involve any training. When combined with fine-tuning (SeTAR+FT), it operates in a 1-shot setting, utilizing only **1,000 images** for training, with just **1.6% of the parameters** being trainable over **5 epochs**. - For comparison, SeTAR achieves an FPR95 of 33.12 and an AUROC of 91.32 without any training. With minimal fine-tuning (SeTAR+FT), it achieves an FPR95 of 32.19 and an AUROC of 92.31 with only 1,000 training samples. Given the significant differences in the amount of training data, the number of trainable parameters, and the computational resources required, we believe that a direct comparison between CLIPN and our method is not entirely fair. Despite the minimal computational resources involved, SeTAR and SeTAR+FT achieve performance levels close to those of CLIPN, demonstrating the efficiency of our approach. # 2. Comparisons with NegLabel - **Difference from NegLabel:** Our method differs from NegLabel [2] in its focus. SeTAR primarily aims to enhance the model's intrinsic performance through SVD pruning, without incorporating any additional knowledge or inputs. In contrast, NegLabel improves OOD detection by constructing large scale virtual negative labels from the data perspective. - **Compatibility with NegLabel:** Since SeTAR and NegLabel have different focuses, they are not mutually exclusive. As mentioned in our paper, SeTAR is highly compatible with various score functions and can also work alongside data augmentation methods like NegLabel. For instance, by simply merging negative labels into the label space for searching and testing, SeTAR can further surpass NegLabel's performance. |ViT-B/16 |iNaturalist FPR↓|iNaturalist AUC↑|SUN FPR↓|SUN AUC↑|Places FPR↓|Places AUC↑|Texture FPR↓|Texture AUC↑|Average FPR↓|Average AUC↑| |:---|-:|-:|-:|----:|-:|-:|-:|-:|-:|-:| |CLIPN\*|23.94 |95.27|26.17|93.93|33.45|92.28|40.83|90.93|31.10|93.10| |NegLabel\* |1.91 |99.49|20.53|95.49|35.59|91.64|43.56|90.22|25.40|94.21| |SeTAR|0.15 |99.54|19.06|95.84|30.63|92.22|42.54|90.30|**23.09**|**94.48**| > \* cited from [2] In summary, SeTAR demonstrates strong performance through its structural enhancements, particularly with SVD pruning, which significantly improves model robustness and OOD detection without requiring additional training. Furthermore, when combined with data augmentation methods like NegLabel, SeTAR's effectiveness is further amplified, showing even greater improvements in OOD detection metrics. --- **References**: - [1] CLIPN for Zero-Shot OOD Detection: Teaching CLIP to Say No - [2] Negative Label Guided OOD Detection with Pretrained Vision-Language Models

Reply to further questions

We sincerely appreciate the thoughtful feedback provided by the reviewer. Here are our responses: # 1. Alternative Loss Function for Eq. 11 - Our loss function (Eq. 11) leverages local features because it requires pseudo-OOD features, which are inherently present in ViTs and can also be constructed in CNNs. In CNNs, alternative methods for OOD feature construction are available, such as the approach proposed in [NPOS](https://arxiv.org/pdf/2303.02966)[4], where boundary ID embeddings are selected based on non-parametric k-NN distances, and outliers are synthesized by sampling from a multivariate Gaussian distribution centered around these boundary embeddings. Additionally, [CLIP-OS](https://arxiv.org/pdf/2404.00323)[5] suggests using CLIP for outlier synthesis, which can similarly be adapted for constructing OOD features in CNNs. - However, due to the complexity of implementation and time constraints, we were unable to conduct the corresponding experiments. Nevertheless, our method is not limited to CLIP models; by utilizing different OOD feature construction methods, our approach can be readily adapted to CNN models as well. # 2. Comparisons on Swin Backbone - We conducted experiments using the Swin backbone with ReAct [3], DICE [1], and ASH [2] methods. Specifically: 1. **Codebase:** For ASH and ReAct, we implemented Swin-transformer based on the [official ASH repository](https://github.com/andrijazz/ash/blob/main/config/vit_config.yml), strictly following the original settings and applying sparsification before the final linear layer. Since ASH does not provide an implementation for DICE, we based our implementation on the [DICE official repository](https://github.com/deeplearning-wisc/dice/tree/master/models). 2. **Hyperparameter Search:** Following ASH, we experimented with various parameter combinations and reported the best results: DICE pruning thresholds included {10%, 15%, 70%}, ReAct clipping thresholds {1.0, 1.5, 1.33}, and ASH-S pruning thresholds included {60%, 65%, 90%, 95%}. 3. **Datasets:** We used ImageNet1K as the ID dataset. - The results are shown below. It is evident that compared to sparsification-based methods, SeTAR exhibits the best performance. ReAct shows improvements over the baseline in both scoring functions, while DICE shows improvement only with MSP; ASH-S performs poorly across the board. | SwinV2-Base| Score| iNaturalist FPR↓ | iNaturalist AUC↑ | SUN FPR↓ | SUN AUC↑ | Places FPR↓ | Places AUC↑ | Texture FPR↓ | Texture AUC↑ | Average FPR↓ | Average AUC↑ | |:-|:-|-:|-:|-:|-:|-:|-:|-:|-:|-:|-:| | Vanilla* | MSP|44.78 |89.89 |63.12 |82.81 | 67.07 | 81.45 |62.04 |82.33 |59.25 |84.12 | | React* | MSP|42.98 |90.39 |61.34 |83.89 | 65.11 | 82.64 |61.22 |81.37 |57.66 |84.57 | | DICE*| MSP|43.02 |89.03 |62.22 |78.31 | 65.82 | 79.35 |57.75 |81.48 |57.20 |82.04 | | ASH-S* | MSP|53.21 |78.72 |73.71 |66.56 | 79.75 | 60.75 |60.27 |75.52 |66.73 |70.39 | | SeTAR (Ours) | MSP| **41.44** |**91.08** |**60.05** |**85.04** | **64.31** | **83.70** |**58.39** |**83.26** |**56.05** |**85.77** | | Vanilla* | Energy |57.52 |81.60 |71.98 |72.93 | 76.90 | 68.90 |53.65 |80.96 |65.01 |76.10 | | React* | Energy |41.78 |88.34 |60.98 |79.19 | 68.07 | 75.56 |53.72|80.39|56.14|80.87| | DICE*| Energy |64.45 |74.91|83.04|62.67|95.18|47.05|94.65|33.45|84.33|58.18| | ASH-S* | Energy |99.67 |19.26 |99.29 |22.91 | 99.51 | 21.49 |98.51 |33.45 |99.25 |24.28 | | SeTAR (Ours) | Energy |**41.71** |**89.42** |**56.53** |**83.29** | **62.84** | **80.20** |**45.37** |**84.76** |**51.61** |**84.42** | > \* denotes our rerun These results demonstrate that SeTAR significantly outperforms sparsification-based methods, highlighting the effectiveness of our approach. # 3. Verification of ASH-S Implementation - Due to the poor performance of ASH-S, we verified the results using the official code. Although the paper does not report ViT results, related scripts are available in their codebase. We tested the official [ViT script](https://github.com/andrijazz/ash/blob/main/config/vit_config.yml), and the results are shown below, confirming that ASH-S continues to perform poorly on ViT. | ViT-B/16 | Score| iNaturalist FPR↓ | iNaturalist AUC↑ | SUN FPR↓ | SUN AUC↑ | Places FPR↓ | Places AUC↑ | Texture FPR↓ | Texture AUC↑ | Average FPR↓ | Average AUC↑ | |:-|:-|-:|-:|-:|-:|-:|-:|-:|-:|-:|-:| |Vanilla*| Energy |64.08 |79.24|72.77|70.25|74.30|68.44|58.46|79.30|67.40|74.31| |ASH-S*| Energy |99.98 |7.28 |99.64|17.82|99.59|19.72|98.09|27.31|99.32|18.03| > \* denotes our rerun **References**: - [1] Dice: leveraging sparsification for out-of-distribution detection. - [2] Extremely simple activation shaping for out-of-distribution detection. - [3] ReAct: out-of-distribution detection with rectified activations. - [4] Non-Parametric Outlier Synthesis. - [5] CLIP-driven Outliers Synthesis for few-shot OOD detection.

Results on CNNs

We sincerely appreciate the reviewer’s insightful comments. Here are our detailed responses: # 1. Results on ResNet50 1. **Setup:** We conducted experiments using only the ID loss, applying low-rank approximation on the in-feature and out-feature dimensions of the convolutional layers, combined with ASH for search. The results are as follows: | ResNet50 |iNaturalist FPR↓ |iNaturalist AUC↑ |SUN FPR↓ |SUN AUC↑ |Places FPR↓ |Places AUC↑ |Texture FPR↓ |Texture AUC↑ |Average FPR↓ |Average AUC↑ | |:--|---:|---:|---:|---:|-:|-:|--:|--:|--:|--:| | Softmax \* |54.99|87.74 |70.83 |80.86 |73.99 |79.76 | 68.00 | 79.61 | 66.95 | 81.99 | | Energy \* |55.72|89.95 |59.26 |85.89 |64.92 |82.86 | 53.72 | 85.99 | 58.41 | 86.17 | | ReAct \* |20.38|96.22 |24.20 |94.20 |33.85 |91.58 | 47.30 | 89.80 | 31.43 | 92.95 | | DICE \*|25.63|94.49 |35.15 |90.83 |46.49 |87.48 | 31.72 | 90.30 | 34.75 | 90.77 | | ASH-P \* |44.57|92.51 |52.88 |88.35 |61.79 |61.79 | 42.06 | 89.70 | 50.32 | 89.04 | | ASH-B \* |14.21|97.32 |22.08 |95.10 |33.45 |92.31 | 21.17 | 95.50 | 22.73 | 95.06 | | ASH-S \* |11.49|97.87 |27.98 |94.02 |39.78 |90.98 | 11.93 |97.60| 22.80 | 95.12 | | SeTAR|10.08|98.11|27.68|94.15|39.22|91.24| 12.54 | 97.51 | **22.38** | **95.25**| > \* cite from "Extremely simple activation shaping for out-of-distribution detection." 2. **Not Highly Dependent on OOD Loss (Eq. 11):** Even under this simple setup, SeTAR outperforms ASH's best performance. This demonstrates that our method can effectively enhance the model's ability to distinguish between ID and OOD samples, even when relying solely on the ID loss. It’s important to note that this result was obtained under a basic setup, and due to time constraints, we did not further tune or explore other detailed configurations specific to CNNs, such as: - **Low-Rank Settings for Convolutional Layers:** The optimal low-rank structure and dimensions for convolutional layers have not been thoroughly researched. For example, ELRT [1] proposes low-rank approximation directly in the high-order tensor format, while other methods [2][3][4] conduct and maintain the 4-D convolutional layer in the format of a low-rank 2-D matrix. - **Pseudo-OOD Feature Extraction in CNNs:** As mentioned in our previous response, there are methods to construct OOD features within CNNs as well, which could further improve our model's performance.

Results on CNNs (Continued)

# 2. Effectiveness of Our Method Across CLIP, Swin, and CNN Architectures 1. **Broad Effectiveness Across Architectures:** Our method has consistently proven effective across a range of architectures. As detailed in Table 1 and Table 7 of our paper, **SeTAR outperforms vanilla methods on both CLIP and Swin models**. Furthermore, even with a basic setup on ResNet, **SeTAR surpasses current state-of-the-art sparsification-based methods**. The comparison with Swin in our previous response underscores the limitations of previous sparsification-based approaches, which struggle with models like Swin. In contrast, **SeTAR achieves state-of-the-art performance across all major vision architectures**, demonstrating its versatility and generalizability. 2. **Significance for ViT-based Models:** ViT-based models, like CLIP, are receiving increasing attention in research due to their scalability and strong performance [5][6][7][8][9]. Our method’s superior results on ViT models highlight its potential for advancing OOD detection in these architectures, making it particularly relevant as ViT models become more widely adopted. 3. **Limitations of Sparsification-Based CNN Methods:** Sparsification-based CNN methods like ReAct and ASH cannot be applied as a post-hoc method to CLIP-based zero-shot OOD detection model. Both methods rely on the assumption that ID and OOD images produce distinct activations in models trained specifically on ID data, such as ResNet50. However, in large-scale pretrained models like CLIP, the activations for ID and OOD images are not significantly different. Consequently, methods like ReAct and ASH, which are limited to models trained on downstream ID-domain tasks, constrain their effectiveness in enhancing CLIP’s zero-shot OOD detection capabilities. In contrast, our method can be applied as a post-hoc method to enhance CLIP's zero-shot OOD detection capabilities. 4. **Contribution of Specialized Methods:** Despite their limitations, specialized methods play a crucial role in their respective domains. For instance, sparsification-based CNN methods significantly enhance OOD detection in CNN models, even though they may not perform well on CLIP models. Similarly, methods like GL-MCM [10] and LoCoOp [11], which utilize CLIP’s local features, substantially improve MCM scores and the performance of fine-tuned models. Although these methods are specialized, they contribute meaningfully to the ongoing development and advancement of the field. --- **References:** - [1] ELRT: Towards Efficient Low-Rank Training for Compact Neural Networks - [2] Learning Low-rank Deep-Neural Networks via Singular Vector Orthogonality Regularization and Singular Value Sparsification - [3] Training CNNs with Low-Rank Filters for Efficient Image Classification - [4] Convolutional Neural Networks with Low-Rank Regularization - [5] Multimodal Learning with Transformers: A Survey - [6] Self-Supervised Multimodal Learning: A Survey - [7] The Llama 3 Herd of Models - [8] Scaling Vision Transformers to 22 Billion Parameters - [9] How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites - [10] Zero-Shot In-Distribution Detection in Multi-Object Settings Using Vision-Language Foundation Models - [11] LoCoOp: Few-Shot Out-of-Distribution Detection via Prompt Learning

Clarification and Innovations in SeTAR for OOD Detection

We sincerely appreciate the reviewer’s thoughtful feedback and the opportunity to clarify and expand on certain aspects of our work. # 1. Clarification on LASER We appreciate the reviewer’s reminder regarding Section 5.1 of the LASER paper. We also took note of this section during our review. What we intended to highlight in our original rebuttal is that, while LASER employs a single greedy search strategy, our work delves deeper into the nuances of conducting greedy search effectively for OOD detection (as detailed in Section 4.4). Specifically, LASER’s approach focuses on composing reductions across layers without exploring the broader landscape of greedy search strategies. In contrast, we systematically analyze and compare different greedy search techniques, evaluating their effectiveness across various layers and backbones. This detailed exploration allows our method to be more finely tuned for the specific challenges of OOD detection, thereby providing a more robust and versatile solution. # 2. Innovation in Post-Hoc Sparsification Methods Post-hoc sparsification methods are widely utilized to enhance CNN-based OOD detection, with well-known examples including ReAct[1], ASH[2], and Dice[3]. These methods typically operate in the weight or activation space, aiming to improve OOD detection by modifying the model’s internal structures. However, a significant limitation of methods like ASH and ReAct is that they are designed for models that have been trained on in-domain (ID) data, such as ResNet50, where distinct activations for ID and OOD samples are expected. These methods fail to generalize to zero-shot OOD detection models like CLIP, where the model has not been fine-tuned on any ID data, and the activations for ID and OOD samples share similar distributions. Our approach addresses these limitations by introducing two key innovations: first, we operate within the space derived from SVD decomposition, which allows us to capture the most informative components of the model while discarding noise. Second, our method is specifically designed to function as a post-hoc approach compatible with CLIP’s zero-shot OOD detection capabilities. This dual innovation not only enables our method to bypass the limitations of traditional sparsification techniques but also allows it to enhance OOD detection in models that are not specifically trained on ID data. We will ensure that these distinctions and innovations are clearly articulated in the revised version of our paper. --- **References**: - [1] ReAct: out-of-distribution detection with rectified activations. - [2] Extremely simple activation shaping for out-of-distribution detection. - [3] Dice: leveraging sparsification for out-of-distribution detection.