Recent advancements in dense out-of-distribution (OOD) detection have primarily focused on scenarios where the training and testing datasets share a similar domain, with the assumption that no domain shift exists between them. However, in real-world situations, domain shift often exits and significantly affects the accuracy of existing out-of-distribution (OOD) detection models. In this work, we propose a dual-level OOD detection framework to handle domain shift and semantic shift jointly. The first level distinguishes whether domain shift exists in the image by leveraging global low-level features, while the second level identifies pixels with semantic shift by utilizing dense high-level feature maps. In this way, we can selectively adapt the model to unseen domains as well as enhance model's capacity in detecting novel classes. We validate the efficacy of our proposed method on several OOD segmentation benchmarks, including those with significant domain shifts and those without, observing consistent performance improvements across various baseline models. Code is available at ${\href{https://github.com/gaozhitong/ATTA}{https://github.com/gaozhitong/ATTA}}$.
Paper
Similar papers
Peer review
Summary
The manuscript considers dense OOD detection under domain shift. The manuscript shows that the contemporary methods for dense OOD detection experience performance drop under domain shift and propose an adaptation framework to mitigate the issue. The proposed framework has two steps. The first step determines whether domain shift exists and attempts to reduce it by adapting the statistics of batch normalization layers. The second step iteratively adapts network parameters by optimizing outlier-aware self-supervised loss. Outlier identification during the test-time training is done by renormalizing arbitrary OOD scores. The resulting method can accommodate various anomaly detectors and achieves competitive results in considered benchmarks for dense OOD detection with and without domain shift.
Strengths
S1. The road driving scenes indeed experience domain shift. E.g. changes in weather conditions and geolocation affect the captured scenes. Hence, the task of dense OOD detection under domain shift makes sense and appears to be novel. S2. The developed method can accommodate multiple methods for dense OOD detection, which advocates for general applicability. S3. The method achieves competitive results in OOD detection in road driving scenarios under considered domain shifts.
Weaknesses
W1. Contemporary works for dense predictions in traffic scenes consider four main types of domain shift: geolocation, weather conditions [a], day-to-night [b], and synthetic to real [c]. Yet, the manuscript deals only with dense OOD detection under different geolocation (RoadAnomaly). Moreover, FS Static-C contains blurred and colour-jittered images which may not adequately reflect real-world domain shifts in traffic scenes. Authors should experimentally cover all possible domain shifts in traffic scenes. W2. The proposed framework requires gradient-based optimisation (and hence backpropagation) during the inference. The manuscript should report the time and memory overhead as in [14,21]. Adding considerable computational burden may make the proposed method inapplicable to the considered application. W3. The proposed framework may only work with models which have batchnorm layers (Sec. 3.3). The manuscript should reflect on models which do not use batch norm layers (eg. attention-based architectures [d]). W4. The manuscript misses relevant works [e,f,g]. [a] Christos Sakaridis, Dengxin Dai, Luc Van Gool: Semantic Foggy Scene Understanding with Synthetic Data. Int. J. Comput. Vis. (2018) [b] Christos Sakaridis, Dengxin Dai, Luc Van Gool: Guided Curriculum Model Adaptation and Uncertainty-AwareEvaluation for Semantic Nighttime Image Segmentation. ICCV 2019. [c] Lukas Hoyer, Dengxin Dai, Luc Van Gool: DAFormer: Improving Network Architectures and Training Strategies for Domain-Adaptive Semantic Segmentation. CVPR 2022. [d] Robin Strudel, Ricardo Garcia Pinel, Ivan Laptev, Cordelia Schmid: Segmenter: Transformer for Semantic Segmentation. ICCV 2021. [e] Shu Kong, Deva Ramanan: OpenGAN: Open-Set Recognition via Open Data Generation. ICCV 2021 [f] Matej Grcic, Petra Bevandic, Sinisa Segvic: Dense Open-set Recognition with Synthetic Outliers Generated by Real NVP. VISAPP 2021. [g] Chen Liang, Wenguan Wang, Jiaxu Miao, Yi Yang: GMMSeg: Gaussian Mixture based Generative Semantic Segmentation Models. NeurIPS 2022
Questions
C1. More detailed Fig. 2 may improve clarity. C2. Since the manuscript introduces a new task it is very important to establish proper evaluation experiments. How does the model perform under different weather and illumination? E.g. when applying transformations similar to [a] which imitate rain/fog. C3. The SMIYC benchmark [3] contains images with different illumination (night) and weather (snow, fog). How does the method perform on this benchmark?
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
2 fair
Presentation
2 fair
Contribution
3 good
Limitations
The manuscript did not reflect on the limitations of the model. One possible limitation may be described in W3.
Post rebuttal
The presented response addresses most of my concerns (W2, W3, W4.) and dense OOD detection under domain shift is an attractive research topic. However, the current state of the manuscript does not consider domain shifts in isolation which is common in the field of domain adaptation e.g. [1]. On the contrary, RoadAnomaly mixes images with different domain shifts. The dataset should be sorted according to the domain (and even enlarged) to show more informative results. [1] Christos Sakaridis, Dengxin Dai, Luc Van Gool: ACDC: The Adverse Conditions Dataset with Correspondences for Semantic Driving Scene Understanding. ICCV 2021. I will increase my score to 4, but I still believe the manuscript should undergo a major revision and another round of reviews.
Response to Reviewer XNCt
Thank you for your comment and for recognizing the appeal of dense OOD detection under domain shift. Regarding your concern, we would like to clarify that our primary focus is on enhancing OOD detection across various levels of domain shifts, rather than adhering strictly to traditional domain adaptation. This approach, which includes consideration of datasets with and without significant domain shifts in isolation, emphasizes our model's practical ability to cope with different degrees of shifts. We believe this perspective is more aligned with the essential and real-world concerns of OOD detection. Concerning the isolation of different types of domain shifts, we appreciate the reference to datasets like [1] that categorize shifts. Yet, in current dense OOD detection datasets, isolating specific domain shifts is a practical challenge. This complexity stems from the lack of explicit consideration of domain shifts during dataset construction. For instance, in datasets like Road Anomaly, various shifts may be intermingled within an image—such as different road conditions with diverse weather—making clear division difficult. To provide information on our model's behavior under various domain-shift types, we've included visualizations in the manuscript (Fig. 2) and the Attached PDF (Fig. 2), demonstrating our method's stable performance across different scenarios. In response to your suggestion, we also conduct additional experiments by introducing smog, color shifting, and Gaussian blur individually to the original FS Static dataset. Results, as shown in Table E3, reveal consistent improvements by our method across isolated domain shifts. We share the hope for future datasets with clearly defined domain shifts and novel classes to further study this evolving field of OOD detection. However, such exploration is beyond the scope of this paper, and we look forward to benchmarks that will facilitate research in this vital area. Table E3. We modify the original FS Static dataset by introducing fog, color shifting, and Gaussian blur separately, to analyze model performance on isolated domain shifts. We compare our method with the previous OOD detection method, PEBAL. | | Fog | | | Color| | | Blur | | | |--------------|-------|-------|-------|----------------|-------|-------|---------------|-------|-------| | | AUC $\uparrow$ | AP $\uparrow$ | FPR95 $\downarrow$ | AUC $\uparrow$ | AP $\uparrow$ | FPR95$\downarrow$ | AUC $\uparrow$ | AP $\uparrow$ | FPR95$\downarrow$ | | PEBAL | 48.37 | 1.58 | 91.82 | 98.58 | 81.93 | 6.26 | 99.46 | 89.46 | 2.07 | | PEBAL + Ours | 98.92 | 79.98 | 3.48 | 99.15 | 87.43 | 2.91 | 99.55 | 90.73 | 1.71 |
On domain shifts
The additional response, in particular Table E3 alleviated my concerns. I hope the analysis of domain shifts in isolation is included in the final version of the manuscript. I increased the score to 5.
Response to Reviewer XNCt
Thank you for your response and for increasing the score. We appreciate your insights and will include the analysis of domain shifts in isolation in the revised manuscript.
Summary
This paper proposes ATTA (Anomaly-aware Test-Time Adaptation), which introduces test-time domain adaptation (TTA) for anomaly segmentation. As a result, anomaly segmentation can be performed well even in a harsher environment where domain shift and semantic shift occur simultaneously. To create an environment with domain shift, the authors create the FS Static -C (Corrupted FS Static) dataset, and develope a Selective Test-Time Batch Normalization (SBN) method to propose a new TTA. They also introduce self-supervised learning using GMM.
Strengths
1. The authors show existing methods are vulnerable to domain shift by creating the FS Static -C dataset and experimenting with it. 2. The effectiveness of TTA in the field of anomaly segmentation is demonstrated experimentally (Table 1). 3. ATTA with SBN and self-supervised learning via GMM, shows better performance than TBN or Tent, which were used in the existing vision field. 4. The authors provide a mathematically convincing motivation.
Weaknesses
1. Table 1 shows that ATTA is exposed to the FS Static -C dataset at test time, and the FPR is re-measured for FS Static -C and the original dataset (FS Static). Therefore, the OOD that must be classified should be seen by the model, so the result should be good. As FS Static -C is a variation of FS Static, it can have a significant impact on the performance of the original dataset. In order to make a meaningful comparison, at least other methods such as Meta-OoD should also be exposed to the FS Static -C dataset and then compared. 2. The contribution and effect of ATTA is unclear. Using OOD data to improve detection performance has been proposed in the past (e.g., Outlier Exposure[1], Meta-OoD[2], etc.). ATTA also eventually exposes the model to additional OOD data to improve detection performance. However, it is necessary to add a description of the advantages that ATTA has over existing methods by exposing it to the test time. For example, it can be shown in a graph that the segmentation performance increases as the batch progresses during the test time. It is also necessary to show how much the performance is improved through TTA on other datasets other than FS Static -C. 3. It is unclear whether a fair comparison was made with the OOD data using methods (e.g., Meta-OoD, Synboost, DenseHybrid) in Table 2. Since TTA is able to obtain more information than MSP, Entropy, Mahalanobis and post hocs, ATTA should be superior to these methods that do not use additional OOD data. Therefore, the main competitors of ATTA are methods that utilize additional OOD data in learning (e.g., Meta-OoD, Synboost, DenseHybrid). The authors' method ATTA uses the FS Static -C dataset as an OOD data for training, but the OOD data used for training in the existing methods mentioned above is not specified. Therefore, it is necessary to perform an ablation study to determine whether the superior performance of ATTA is the effect of ATTA, or simply the effect of data augmentation caused by the introduction of the FS Static -C dataset. In addition, ATTA adopted PEBAL as a partner and compared the performance of PEBAL, which was trained on COCO as an additional OOD, with the existing report. PEBAL should also be compared after being exposed to the FS Static -C dataset (excluding ATTA). 4. The metrics used are AP, AUROC, and FPR95. However, except for AP (Average Precision), these metrics are mainly used in OOD classification rather than anomaly segmentation. It is also necessary to compare the sIoU, PPV, and F1 metrics proposed in the benchmark paper [3] mentioned by the authors in related works. 5. As the authors mentioned in the Conclusion, TTA is used, so the learning is performed simultaneously at the test time, so the inference time is slow. It is necessary to check if the overhead is too large by comparing the inference time for each method. In particular, GMM clustering must be performed for each image, and it is necessary to check if the overhead due to this is excessive. [1] Hendrycks, Dan, Mantas Mazeika, and Thomas Dietterich. "Deep anomaly detection with outlier exposure." arXiv preprint arXiv:1812.04606 (2018). [2] Chan, Robin, Matthias Rottmann, and Hanno Gottschalk. "Entropy maximization and meta classification for out-of-distribution detection in semantic segmentation." Proceedings of the ieee/cvf international conference on computer vision. 2021. [3] Chan, Robin, et al. "Segmentmeifyoucan: A benchmark for anomaly segmentation." arXiv preprint arXiv:2104.14812 (2021).
Questions
1. Is the FS Static -C dataset is just a data augmentation of the original FS Static dataset? Or is there a significant difference enough to be called another 'dataset'? 2. Is there a performance change when ATTA is trained on the original FS Static dataset rather than the FS Static -C dataset? 3. Is there a performance change when K-means is used instead of GMM clustering to separate inlier and outlier clusters from the OOD score set G for self-supervised learning?
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
2 fair
Presentation
3 good
Contribution
2 fair
Limitations
1. The authors have well summarized the common features of the OOD field that are affected by the performance of the backbone, and the weakness that the inference is slow because learning is performed during the test time when TTA is used. 2. The inference time is expected to be slow, so it is necessary to compare the inference time for each method to check if the overhead is too large.
Summary
This paper focuses on the challenging task of open-set semantic segmentation (i.e., dense out-of-distribution (OOD) detection) with domain shift. It proposes a dual-level test-time adaptation framework to overcome domain shift and semantic shift simultaneously, which leverages low-level feature statistics to detect whether domain shift exists while identifying pixels with semantic shift by utilizing dense high-level feature maps. Specifically, it designs an anomaly-aware self-training component to address potential domain shift and improve its ability to detect novel classes through re-balanced uncertainty minimization. The proposed framework is demonstrated to obtain consistent performance improvements across various baseline models.
Strengths
1. This paper explores the realistic and challenging task of open-set semantic segmentation in a real-world scenario with domain shift, considering the impact of both domain shift and semantic shift (novel class from unknown domain) comprehensively. 2. The method in this paper seems reasonable and the experimental results prove the significant superiority of the proposed framework on several OOD segmentation benchmarks, regardless of with or without domain shifts. 3. The method and math presentation in this paper is generally clear.
Weaknesses
1. The visualization of the method in Fig.2 appears overly simplistic and fails to highlight the key components of the proposed framework effectively. 2. How to choose the parameters of the GMM in L220? Is the performance sensitive to their variations? The hyperparameter experiment of these parameters would be desirable. 3. Fig.1 intuitively demonstrates a "domain shift" (fog), but through the visual experiment in Fig.3, I can not understand what the specific "domain shift" that exists in the Road Anomaly dataset for the semantic segmentation is. This casts doubt on the practical application of this paper, despite the experimental evidence confirming the effectiveness of the proposed framework. 4. All the formulas in this paper have not been correctly punctuated.
Questions
Please refer to the weakness part.
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
Do not apply.
Summary
The papers deal with two levels of domain shift in semantic segmentation, namely the domain shift on the semantic pixel level and the domain shift on the image level. The paper argues that the current dense out-of-distribution (OOD) detection methods are particularly vulnerable in the presence of image-level domain shifts and therefore likely to predict wrong OOD scores on the pixel level. Based on this observation, it presents a two-stage test-time adaptation approach. In the first step, it is performed selective test-time batch normalization, forcing more adaptation in the scenarios which are identified to be novel, while n the second step, it is proposed a self-training procedure using an anomaly-aware output map to enhance the dense OOD detection. The approach is evaluated on a realistic domain-shift dataset, as well as, two non-domain-shift datasets where it shows good performance. While the rebuttal addresses several points of the reviews, the changes and additional experiments discussed are several. The paper should go through another round of peer review with many changes. Therefore, I will not increase my score.
Strengths
- The analysis of how current OOD detection methods perform on datasets with distribution shifts, as shown in Figure 1 and Table 1, is very valuable and shows the importance of the problem studied throughout the paper. - The idea is easy to follow. - The combination of two steps, first to reduce the domain gap at the image level and second to improve the dense OOD detection performance of the models, is not well explored in the literature. - The results are convincing when compared to the prior work.
Weaknesses
- The clarity of the method has place for improvement. For instnace, the notation is one point, e.g. 120 line x^s, x^s not defined, Eq. 1 \mathcal{N} not defined. The method can be understoood in general, it could be further polished to make it easier. There are also minor typos, e.g. line 38, line 245, line 194. Also, KL divergence is not clearly defined in Eq 1. - In Fig. 1: What is shown in (a) and (b)? A detailed description would be helpful. - For comparison with their method, the authors add Tent to PEBAL. An explanation of how this is implemented would be helpful. - Table 1 compares 6 different OOD detection methods on the synthesised corrupted FS Static (FS Static -C) dataset. It shows a performance drop for all OOD detection methods on the FS Static -C dataset, but applying the proposed approach to PEBAL reduces this drop. It would be interesting to see if a similar effect can be achieved for the remaining 5 OOD detection methods.
Questions
- It is not straightforward to implement the paper. There is no discussion about releasing the code.
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
3 good
Presentation
2 fair
Contribution
3 good
Limitations
- Not applicable .
While the rebuttal addresses several points of the reviews, the changes and additional experiments discussed are several. The paper should go through another round of peer review with many changes. Therefore, I will not increase my score.
Response to Reviewer TGMY
Dear Reviewer TGMY, Thank you for your thoughtful feedback. Regarding the extent of the changes, we'd like to emphasize that our rebuttals are targeted responses to the comments raised by reviewers. Rather than altering the main content of our paper, they are intended to complement and reinforce our existing arguments. Therefore, those additional materials will primarily be placed in the appendix and should not affect the paper's main structure. Here is a summary of our additional results and changes: **Summary of the Additional Results:** - **SMIYC Benchmark (General Response 1):** These results enhance our testing on the domain-shift dataset, aligning with existing findings on FS Static -C and the Road Anomaly datasets. - **Lost and Found Dataset / online FS test set (Response to Reviewer Nknk):** These complementary results align with existing findings on the Fishyscapes Lost&Found and Fishyscapes Static offline datasets. - **Combination with Other OOD Detection Methods on FS Static -C (Response to Reviewer TGMY):** The results demonstrate the general applicability of our method which is consistent with our existing findings shown in Table 2. - **Overhead Analysis (General Response 2):** Additional inference time has been discussed as a potential limitation in our conclusion part. While efficiency is not our paper's focus, we have included these results to address concerns. - **Combination with Architectures without BN (Response to Reviewer XNCt):** Our paper primarily focuses on architectures with BN. The experiments with Segmenter (using Layer Norm) demonstrate extendibility to other architectures but are not a central focus of our paper. **Summary of Other Changes:** - **The Detailed Figure 2:** This enhanced version is already provided in the attached PDF. It can directly be replaced with the original Figure 2, with minimal impact on other parts. - **Notation Clarity, Minor Typos, Detailed Caption, Formula Punctuated (Response to Reviewer TGMY, 7M1E):** These are minor points and will not change the overall content. As outlined above, the additional results and changes are designed to support and reinforce our paper without altering its main content. We appreciate your thoughtful insights and remain open to any further suggestions. Sincerely, Authors
Summary
The proposed method considers the OOD sample with domain shift and semantic shift. They address the problem of current OOD approaches that often ignore to handling domain shifts. They introduce an anomaly-aware test-time adaptation method that jointly tackles domain and semantic shifts. The experiments on different benchmarks, the proposed method demonstrated significant performance gains on various OOD segmentation benchmarks. They show significantly robust performance on benchmarks with notable domain shifts.
Strengths
The paper has well-organized writing and clear motivation for each part of the proposed method. The proposed method is relatively novel and presented clearly. The proposed method achieves SOTA results with large performance gains compared with other SOTA methods. The proposed method first time focuses on handling domain shift OOD detection, which has been overlooked by previous OOD methods.
Weaknesses
From Figure 2, readers cannot figure out any details regarding each of the proposed components and the overall framework. I would suggest the authors could include more details in the figure to have more detailed basic information for each of their contributions. Although the authors compare their approach to the Road Anomaly dataset for domain shift effectiveness evaluation. The domain shift is even more significant in SegmentIfYouCan benchmarks. Hence, I would suggest the authors to show more results on the SegmentIfYouCan benchmark and see if the proposed ATTA can improve the results. From what I observed, the performance of previous SOTA such as PEBAL, and Meta-OOD, perform relatively worse on the domain-shifted benchmarks like SegmentIfYouCan. It would be better if the authors can compare their approach to the online Fishyscape testing set. The validation set only contains a few images, which may not be enough to effectively quantify the performances of the method. Computational efficiency is important to perform real-time detection for self-driving systems, could authors present the training and inference time of the proposed approach and compare it with other methods?
Questions
The computation efficiency of the proposed method is unknow.
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
4 excellent
Contribution
4 excellent
Limitations
According to the paper, the model's improvement is less noticeable with weaker backbones. The proposed method will add additional time.
It appears that I initially misunderstood the utilization of the FS Static-C dataset. I commend the authors for providing clarification on their work in the rebuttal. Upon conducting a thorough re-evaluation of both the main manuscript and the authors' rebuttal, I have come to the realization that the authors have diligently tackled the majority of the concerns I had raised. Consequently, I see no justification for maintaining the initial score, and I am inclined to raise it. Nonetheless, I agree that the paper should go through another round of peer review with many changes. Hence, I will refrain from making a significant increase in my score.
Response to Reviewer C29A
Thank you for re-evaluating our paper and for increasing the score. We would like to clarify that the proposed changes, as detailed in our response to Reviewer TGMY, are specific and minimal. These adjustments are designed to address the comments raised without altering the core of our work, and the results are clearly presented in the rebuttal.
Authors rebuttal
I acknowledge the authors rebuttal and I am encouraging reviewers to reflect on.
Thanks for the iilustration on the concept of "Domain Shift". I have read the responses and all my concerns are addressed. I will adjust the rating accordingly
Online Fishyscapes Testing Set Results
Dear Reviewer Nknk, We have now obtained our results for the online Fishyscapes testing set. As presented in Table A3, our method outperforms the previous state-of-the-art, PEBAL. We appreciate your insights and remain open to further suggestions. Table A3: Results on the Fishyscapes online test benchmark. The results for our model were obtained by submitting it to the benchmark organizer. The results for PEBAL were taken from their published paper. | | Online FS Lost & Found | | |--------------|:---------------:|:----:| | | AP$\uparrow$ | FPR$\downarrow$ | | PEBAL | 44.17 | 7.58 | | PEBAL + ATTA (Ours) | 55.94 | 4.66 | | | Online FS Static | | |--------------|-----------|------| | | AP$\uparrow$ | FPR$\downarrow$ | | PEBAL | 92.38 | 1.73 | | PEBAL + ATTA (Ours) | 94.68 | 0.68 |
Decision
Accept (poster)