To deploy and operate deep neural models in production, the quality of their predictions, which might be contaminated benignly or manipulated maliciously by input distributional deviations, must be monitored and assessed. Specifically, we study the case of monitoring the healthy operation of a deep neural network (DNN) receiving a stream of data, with the aim of detecting input distributional deviations over which the quality of the network's predictions is potentially damaged. Using selective prediction principles, we propose a distribution deviation detection method for DNNs. The proposed method is derived from a tight coverage generalization bound computed over a sample of instances drawn from the true underlying distribution. Based on this bound, our detector continuously monitors the operation of the network out-of-sample over a test window and fires off an alarm whenever a deviation is detected. Our novel detection method performs on-par or better than the state-of-the-art, while consuming substantially lower computation time (five orders of magnitude reduction) and space complexities. Unlike previous methods, which require at least linear dependence on the size of the source distribution for each detection, rendering them inapplicable to ``Google-Scale'' datasets, our approach eliminates this dependence, making it suitable for real-world applications.
Paper
References (61)
Scroll for more · 38 remaining
Similar papers
Peer review
Summary
This paper studies the topic of model monitoring through the lenses of input distribution shift detection during test time. They propose a distribution deviation detection method based on selective prediction coverage bound violation counting. They detect data distribution shift based on windows and studies the adaptation of single-instance confidence functions for this task. They compare to baselines such as the KS test and MMD method over the softmax or penultimate layer embeddings, and two single instances baselines based on the maximum softmax response and the output entropy with a t-test. The experimental benchmark is over ImageNet with popular models and shifts based on adversarial attacks, data corruption, and semantically different data.
Strengths
- They present an original perspective on drift detection inspired from selective classification focused on real-world application. - The algorithm exhibits great time complexity, space complexity, and good performance. - Clean theoretical formulation of their algorithm, easy to understand. - Extensive experimental benchmark and comprehensive ablation study on the hyperparameters. - Overall, the paper is well-written and easy to follow.
Weaknesses
- The choice of confidence-rate function might be a bottleneck of the method. Some models can exhibit over-confident predictions even to far OOD data. Relying on the softmax output as a confidence method might present a risk. - The formulation by counting the violations of the coverage bounds is interesting but it lacks motivation. It is hard to understand why computing the coverage bound would be more powerful than the t-tests with the confidence function values for each $x_k$. - Contribution (1) does not seem very useful. Why would someone desire to limit coverage without controlling risk? - In Table 1, the time complexities for MMD and KS might be inflated, as they could be improved by slightly changing the implementation. E.g., for MMD, the term with $O(dm^2)$ time complexity could be computed offline on a fit step. ### Secondary observations In line 375 the authors defined FPR@95TPR but following on in the paper they call it TNR@95TPR. Figure 4 would be easier to read if the AUROC numbers were written in the heatmap instead of the colorbars.
Questions
1. The test set evaluation setup is unclear to me. The testing windows are pure out-of-distribution/corrupted data? What about mixed windows with in-distribution and ood data, which would be a more realistic setting. Have the author tried windows with a mixture of in-distribution and out-of-distribution data? I would strongly encourage adding an experiment in these lines to strengthen the experimental part of the paper. 2. For the case of k=1000, is there a single window? It would mean that you compute the prediction for each method over the 15 seeds and compute the metrics in these predictions? 3. Could the authors elaborate more on the claim that "detection power might lay within lower coverages" (line 362)? To my understanding, lower coverage simply gives a higher (more selective) threshold $\theta$ for the CF, thus, it rejects more less confident samples. 4. Could you explain in more details the novelty of Algorithm 1, Lemma 4.1 and Theorem 4.2 w.r.t references [11,12]? 5. A more open ended less important question. Why do some baselines and your method decrease/oscillate performance with window size on some types of drift?
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
3 good
Presentation
4 excellent
Contribution
3 good
Limitations
The authors do not discuss the limitations of their work in great detail. A greater effort in this direction would be appreciated. For instance, on ImageNet-O, there appears that MMD Embeddings exibits better performance. So maybe the presented method is more appropriated to some types of drifts. Also, I would argue that a variation of 5% in AUROC on the ablation of $\delta$ and $C_{target}$ shows some sensitivity to hyperparameters.
Summary
The paper proposes a window-based distribution shift detection method to detect whether the samples in the window are in-distribution (ID) or OOD. The method takes a window containing samples as input and computes a confidence score using the proposed coverage-based detection algorithm. The method is evaluated on various OOD benchmarks when imagenet is ID dataset. The results demonstrate that the proposed method is computationally more efficient than the existing methods and achieve high accuracies in majority cases.
Strengths
- The idea of detecting OOD samples within a window is interesting and in a different direction than the majority of OOD detection literature. - The paper aims to present a theoretically supported and computationally efficient algorithm for this problem.
Weaknesses
- Lack of intuition and convoluted description of the method: The intuition behind the proposed method is unclear. I didn't quite understand why this method should perform better than the existing ones. The theoretical part should be explained after giving some motivation for the method. - Lack of experiments: The paper presents results only when ImageNet is ID dataset. There should be experiments with different ID dataset to understand if the results generalize to other datasets. - Some experimental details are not clear: It is unclear to me how windows are created in test time. Can a window contain both ID and OOD samples? Since the method labels the whole window as OOD or ID, I wonder how the method's performance changes with different ID and OOD sample ratios within a window.
Questions
Please address my concerns in the weaknesses section.
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
3 good
Presentation
2 fair
Contribution
2 fair
Limitations
Limitations of the method are not discussed. I don't see any potential negative societal impact of the work.
Summary
The paper proposes a methodology to detect distribution shifts in windows (set) of observations for pre-trained deep learning models on computer vision tasks. The method compares the empirical coverage of a selection function to pre-determined computed bounds. During the fit stage, a guaranteed coverage selection algorithm determines C_target different bounds. During inference, the method computes the empirical coverage of the window to compute the total violations to the bounds. Finally, since V follows a normal distribution, a t-test statistic detects distribution shifts. The proposed method requires orders of magnitude less time and memory than the current SotA for large datasets. The paper presents a comprehensive comparison with SoTA methods in ImageNet.
Strengths
- The paper focuses on identifying distribution shifts, a very important problem with multiple applications. - The methodology is appealing and very simple. - The proposed method is theoretically supported. - The results demonstrate the method is faster and requires less memory than the current SoTA. - The paper is well written, and the ideas are clearly motivated. - The experimental setting is comprehensive and well-designed. - Code to replicate results is provided.
Weaknesses
- The proposed method performs better on smaller windows (k~50), which is contrary to the main claims of the paper that the proposed method can leverage population-level information. - One of the main applications to motivate the problem is determining the need to retrain pre-trained models. However, the size of windows (k) used in the experiments seems small for this application, especially since large models are potentially deployed in datasets with millions of observations. - The performance benefits of larger samples (k~1,000) are smaller than existing methods such as KS. - The paper only tests the simple method where V is an average of the C_target scalars. - Coverage is an established metric, and it could be directly used as a distribution-shift detector for the type of application discussed in the paper. It would be interesting to compare the method with a single coverage target value to better understand the benefits of the extension proposed in the paper. Appendix E presents this ablation study but is missing some important information. For instance, what value of C^* is used when C_target=1? - Limitations are not properly discussed. It seems that the accuracy advantages of the method are limited to the less useful setting where k is small.
Questions
- Which target coverage is used in the appendix when C_target=1? - Have authors considered using other datasets than ImageNet?
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
The paper does not properly discuss limitations (see above).
Summary
The paper presents a new method for detecting distribution shifts within a given window of samples. This coverage-based detection algorithm is theoretically motivated and can be applied to pre-trained models. The authors claim the low computational complexity of their framework, and experiments show that the framework outperforms the baselines across a variety of distribution shifts such as pixel-space adversarial examples.
Strengths
1. The paper is well-written with a good storyline that formulates the proposed framework. 2. The window setting is an attractive problem that has yet to be well explored by past works. The target models comprehensively include convolution-based and attention-based. 3. The paper well defines the domain and assumptions of the task, making it clearly comparable to OOD frameworks. 3. The authors motivate the framework by theoretical analysis, as well as conducting empirical experiments that demonstrate the effectiveness of the framework.
Weaknesses
1. From the perspective of pixel-space adversarial attack detection, authors only evaluate the most basic baselines of FGSM and PGD. It will be constructive to the validation of the work if authors conduct additional experiments on the CW attack [1] and GAN perturbations [2]. 2. Authors intend to claim that the framework is effective to detect naturally (semantically) mutated images. However, the authors did not elaborate sufficient validations on the line of research for semantic adversarial examples [3]. It will be beneficial to the work if the authors have discussions on the effectiveness of the work for this line of research. 3. The framework should be evaluated on scalable benchmarks proposed in recent years such as WILDS [4] V1/V2. [1] Towards Evaluating the Robustness of Neural Networks [2] Generating Adversarial Examples with Adversarial Networks [3] Semantic Adversarial Attacks: Parametric Transformations That Fool Deep Classifiers. [4] WILDS: A Benchmark of in-the-Wild Distribution Shifts
Questions
Please address my concerns and questions in the weakness section. To summarize, this paper is a fair contribution to the OOD research. Although the paper is well-structured with a clear storyline, given several major concerns stated in the weakness section, I will rate it as borderline accept at this stage. I will keep track of authors' responses, and consider revising the rating based on the soundness of responses.
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
3 good
Presentation
3 good
Contribution
2 fair
Limitations
The authors discussed potential improvements and future works in section 7.
Summary
A method to detect shifts in the input distribution of deep neural-networks is presented. The method takes a window of samples as input and calculates deviation from a tight coverage generalization bound. Here, coverage is empirically defined as the fraction of input samples that are non-rejected based on a confidence function, i.e. those which are predicted with sufficiently high confidence. The method is evaluated on standard image classification datasets using various commonly used CNN architectures.
Strengths
The method is well motivated and the detection metric ('sum bound violations') is defined and derived in a principled manner. Since the method is based on principles of selective classification, a by-product is a confidence threshold $\theta$ and a corresponding selection function $g_\theta(x)$, which are useful to have in practical situations. What is interesting is that the method does not detection a distribution-shift based on uncertainty or confidence of the input samples, but rather by the deviation of coverage from pre-calculated bounds at several target values based on these uncertainties. Similar to reference [12], the method does not rely on uncertainty values being calibrated.
Weaknesses
1. The derivation of the main steps of the algorithm (Section 4.1) mirrors closely the derivation of the selection-based classifier in Section 3 of the paper cited as reference [12] in the manuscript. This isn't a weakness or limitation in my view, but it is worth noting the close conceptual similarities. 2. What is concerning though is the rather weak experimental evaluation: a. The authors claim that single-instance will not perform as well as their window-based method. However, the only single-instance baselines compared against are softmax response and entropy. These are rather weak baselines and there are several stronger baselines (several of which have been cited by the authors) that should have been compared against. b. It would have been really interesting and useful to see this method applied with different types of uncertainty values apart from softmax (feature-based such as Mahalanobis, predictive uncertainty from Bayesian neural networks, etc.). That would have served as a much stronger evidence of its superiority over other methods such as KS and MMD. Admittedly, it's not possible to cover all aspects in one paper, but this would have been a rich addition in my view. c. It is mentioned that 49000 out of 50000 samples in the imagenet validation set are used as detection training set. I am curious why use the validation data for this purpose. Why not select 49000 samples from the training set of 1 million+ images? 1000 samples to apply the shift to seems rather small. Can the authors please explain this? 3. Somewhat confusing and overlapping notation. $k$ is used for $\lceil \log_2 m \rceil$ in theorem 4.2 and also as size of window $W_k$. Would improve readability if different variables were used for these two different quantities.
Questions
There are questions of a few technical aspect which I'm hoping the authors can help clarify. 1. It is mentioned that in reference 12, "a risk generalization bound was developed for selective classifiers" but that application of that would require labels, but that theirs doesn't. Given the duality between risk- and coverage-based analysis in the two papers (ref. 12 and this manuscript), can the authors provide some more details to support this statement please? 2. The bounds are calculated on a dataset of size $m$ while the test window is of a different size $k$. Should these not be matched? 3. What value of $k$ is used for the visualization in Fig. 2? It would be good to understand the impact of the window-size $k$ on the curves for both the no-shift case and the shift case. Specifically, would the coverage bound and empirical coverage continue to remain nearly identical even when $k$ drops to a low value? 4. Should the inequality in Step 9 of Algorithm 1 be $\geq$ instead of $\leq$?
Rating
5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
2 fair
Presentation
2 fair
Contribution
3 good
Limitations
No societal impact.
Summary
The authors tackle the problem of out-of-distribution detection on a stream of instances. They introduce a method for coverage-based detection (CBD) that creates bounds and thresholds for monitoring by applying a newly derived selective with guaranteed coverage algorithm and then conducts a t-test on the violation sum between a window of newly received samples and a given detection dataset. The method is then analyzed for complexity and evaluated experimentally against ImageNet with various architectures. The method scales up with the size of the detection-training set size given more favourable complexity than previous baselines.
Strengths
1. The case is made that at large scales of data the provided method is more practical for implementation and can match or outperform existing methods for detecting distribution shift on streams of data. 2. The authors provide a wide range of data and analysis to back up their claims: 2.1 Derivation of the SGC and CBD algorithms for estimating coverage and detecting violations. 2.2 Lower complexity 2.3 Experimental validation on ImageNet with various architectures and types of distribution shifts. 2.4 Showing a walkthrough of bound and threshold identification which helps 3. They provide a thorough explanation of their method and how it differs from existing baselines like KS and MMD based tests that are popular both in research and industry. Based on the results here I could easily see CBD replacing either of these methods in existing setups.
Weaknesses
1. Experimental setup:Window-based methods in any task rely quite heavily on the dataset in question. It would have been great to see a more diverse set of benchmarks for out-of-distribution detection if they could be modified to the given problem statement. E.g. Wilds (Koh et. al., 2021) That being said I appreciate the various shifts applied to test on different kinds of distribution shifts. 2. Presentation: I found Table 2 hard to parse and dense. I understand it is hard to fit in all the data but there could be some improvements. 3.
Questions
1. It seems that while the method scales with the size of the detection-training set, it converges with other methods as the window size increases (except with ViT-T architecture). Increasing the window size leads to a bigger jump in evaluations like Detection Error once the window size is above a certain threshold. What is the relationship between m, k with respect to out-of-distribution evaluation and is there any reason why someone might prefer to scale up the window size rather than the detection-training set and thus nullify the complexity advantages of this method?
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
3 good
Presentation
2 fair
Contribution
3 good
Limitations
1. Requiring access to the final layer of weights in the model rather than being fully black-box like some of the other methods. 2. The authors don't highlight any particular limitations for practitioners to watch out for. Even suggestions around tuning the threshold parameters in different data settings could be helpful (or perhaps they are correct that the defaults they use are sufficient for all cases).
Summary
This paper draws attention to the problem of window-based distribution shift detection and introduces a new method for it. The method is theoretically sound and demonstrated empirically to be effective on datasets with multiple types of shifts.
Strengths
- The proposed method is evaluated on extensive datasets with various types of shifts and outperforms selected baselines. - The computational complexity is impressively low. - Due to its lower computational complexity, the proposed method is scalable compared to the baselines.
Weaknesses
There aren't significant weaknesses in my opinion, but several minor ones might need clarification: - I am concerned with the novelty of Algorithm 1, selection with guaranteed coverage, since most parts here are similar to SGRC in [1]. Is coverage the only modification here? - The paper set out with a window-based shift detection task, which feels like a generalization from the single-instance task to me. However, there are no results of $|W_k|=1$. Can the proposed method still outperform baselines in this case? (Authors don't have to add new experiments here). - The goal is to detect if there is a shift in a window $W_k$ of test data. I'd like to know how the proposed method will behave when only a portion of $W_k$ is from a different distribution. [1] "Selective Classification for Deep Neural Networks", 2017
Questions
- In equation 2, what does $|m\cdot \hat c(\theta, S_m)|$ intuitively stand for? Why not just add up to $m$? - Is this typo? line 55: "read-world".
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
See the weaknesses above.
Thanks for providing a rebuttal with fair arguments. I will keep the current rating.
Thanks for the authors' responses. After reading comments from other reviewers, I agreed on several points: - Reviewer **7FwS**'s concern about the limited performance gap with large window size. However, it was noted by authors that detection with a smaller window size implies a "harder" scenario ("The intriguing question, in our opinion, is which method excels for smaller $k$ values"). And also, in my opinion, the scalability of the proposed method could excuse away the narrowing performance gap. - Reviewer **HyPc**'s concern about window size value in Fig 2. The additional visualisation pairs provided in the attachment show consistency with Fig 2. - Authors' comment on my second concern and Reviewer **M7ue**'s first concern help understand the motivation behind the proposed method. Overall, the authors have addressed most of my concerns. Thus I would like to keep my original score to support the acceptance.
Thank you to the authors for a detailed response to all reviewer comments. Much appreciated! A few of my concerns have certainly been addressed, in particular the points about + Why the risk generalization bound cannot be applied (since in its formulation as presented in [12], it requires access to labeled data), + Impact of $k$ on the visualization, for which the authors have provided curves, and + The explanation for why dataset size $m$ and window size $k$ can be different (although, I would prefer a more formal justification). A couple of questions/concerns do remain, however: 1. I do think that a lack of benchmarking against stronger single-samples baselines is a weakness though. 2. I am also not convinced by the authors response that they used validation samples to train their detection module because validation data uncertainty estimates might not be consistent with train data uncertainty estimates. That is true, but isn't that always the case for any machine-learning approach? We never train models using samples validation or test set, and there is always a gap between accuracy of a model on training data and accuracy on test data. Why should this paradigm not apply to training the proposed detection module? Just to clarify, I do understand that a separate hold-out validation set of 49000 samples has been created and that final results have not been reported on this held-out set, but on the remaining 1000 samples (which serve as the final test set). However, I still believe that this should have been created from the training data as only then would one be able to assess if the detection module was capable of detecting between true OOD distribution shifts, and not getting confused by train-vs-val shifts.
Authors response to Reviewer HyPc
Thank you for taking the time to read our response in detail. We will address in full below your remaining concerns. We hope that, given our previous response to all your suggestions and comments and the further clarifications below, you would kindly reconsider your rating. Your consideration is highly appreciated! **"I do think that a lack of benchmarking against stronger single-samples baselines is a weakness though."** As we stated before, we hypothesize that using those stronger single-instance baselines might be impractical in terms of time and space complexity when applied to large windows. Nevertheless, we believe we have a couple of strong arguments for why our method can outperform any suggested single-instance baseline. Let us elaborate. Our method leverages an uncertainty estimator, denoted as $\kappa_f$, which serves as its primary engine (refer to Algorithm 2, $\kappa_f$ is an input). Thus, our approach is versatile -- any uncertainty estimator can be integrated with it. In our experiments, we utilized two specific ones: Softmax-response and Entropy. Therefore, we compared our outcomes with the two corresponding single-instance baselines (Softmax-response and Entropy) -- we consistently outperformed these baselines. We want to point out that single-instance baselines are basically using a coverage of 1 (they operate on each sample), and one of the main claims in our paper is that we are *maximizing detection power through lower coverages*, we support this claim in Section 6.2 and with Fig. 2(b) -- which shows that the largest bound deviation is around a coverage of 0.4 (notably less than 1). In particular, for Fig. 2(b) we used target coverages with values\ {$0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9$} (see lines 219-220 in the paper). Thus, given any single-instance out-of-distribution detector, we can integrate its uncertainty mechanism ($\kappa_f$) to our method, and by adding a target coverage of $1$ to our list of target coverages values\ ({ 1 } $\cup${0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9}), we should match or even surpass its detection performance on windows. **"I am also not convinced by the authors response that they used validation samples to train their detection module, because validation data uncertainty estimates might not be consistent with train data uncertainty estimates. That is true, but ..."** Before diving into the justification for this experimental setting, it's important to highlight that while we could have used the training data to train our detector, doing so would have necessitated simulating the no-shift scenario by drawing 1,000 samples from that same training dataset. First, we would like to emphasize that, we adopted the common experimental approach, also outlined in references [1,2,24,25], where the no-shift scenario is represented by samples not previously encountered by the model. Thus, we also used a separate validation set. The same experimental protocol is also a standard practice when performing calibration, which is a conceptually similar task whereby you are given a trained model and would like to adjust the model's predictions to better align with the true probabilities of the outcomes. In the calibration literature, the validation is often called a calibration set. The use of an independent calibration set is indeed required and is a common practice; see for example references [3,4,5]. To intuitively explain the importance of an independent calibration set imagine you're prepping for an exam using specific questions (the "training set"). If tested on these same questions, you'd excel due to memorization. However, this doesn't mean you'd perform well on new questions (the "test set"). Similarly, in model calibration, training a model on a dataset is like practicing specific questions. Calibrating with the same data is like being tested on practiced questions, leading to seemingly accurate but potentially misleading results. On new data, the model might not be as reliable. In essence, calibrating a model on its training data can give a false sense of its accuracy, risking errors in vital applications. We hope this clarifies the point. We remain available to address any further questions or clarifications the reviewer might have. **References:** [1] - Enhancing the reliability of out-of-distribution image detection in neural networks. ICLR. [2] - Generalized ODIN: detecting out-of-distribution image without learning from out-of-distribution data. CVPR. [3] - What can we learn from the selective prediction and uncertainty estimation performance of 523 ImageNet classifiers? ICLR. [4] - Conformal prediction with missing values. ICML. [5] - Conformalized quantile regression. NeurIPS. [24] - Detecting out-of-distribution examples with gram matrices. ICML. [25] - A simple unified framework for detecting out-of-distribution samples and adversarial attacks. NeurIPS.
**"it's important to highlight that while we could have used the training data to train our detector, doing so would have necessitated simulating the no-shift scenario by drawing 1,000 samples from that same training dataset"** I believe this is the key point that we are differing on. I believe that for training the detector, the data samples should be taken from the training split of the dataset of interest, while to calculate metrics such as AUROC, etc. the in-distribution samples should be drawn from the (unperturbed) test split (validation split in case of ImageNet). This is the protocol that is followed in OOD detection work, even in the references listed in your response. From reference [1]: "At test time, the *test images from CIFAR-10 (CIFAR-100) datasets can be viewed as the in-distribution* (positive) examples." From reference [24]: "For each of these models, we considered the corresponding *test partitions as the in-distribution (positive) examples*." From reference [25]: "We only use *test datasets* for evaluation." (emphasis mine in all the above quotes taken directly from the references). If the in-distribution samples in these papers had been taken from the training set, then their results (AUROC, TPR, etc.) would have been superior compared to what they have reported. Hence, this is a serious issue in my view. However, I am more than willing to be corrected in case my understanding on the situation is wrong. I look forward to your response.
Authors response to Reviewer HyPc, Part 2
After thoroughly reviewing your concerns on this issue, we believe there has been some misunderstanding. Let us start with some definitions, * We are given the partitions $D_{\text{train}}$ and $D_{\text{test}}$ as in ImageNet. To aid the discussion and our experimental objectives, we define these three sets: * $S_{\text{train}}$ : Used for model training (not the detector, the actual neural network). * $S_{\text{val}}$ : Used for the detector's calibration/fitting (specifically for establishing thresholds and bounds in our case). * $S_{\text{test}}$ : Utilized for evaluating detection performance. In addition to this evaluation dataset, we also use for performance evaluation an OOD (Out-Of-Distribution) dataset. This OOD dataset is not relevant to the present discussion. From our current interpretation of your proposition, the suggested setting is: * $D_{\text{train}} = S_{\text{train}} \cup S_{\text{val}}$. * $D_{\text{test}} = S_{\text{test}}$. In our implementation, the setting is: * $D_{\text{train}} = S_{\text{train}}$. * $D_{\text{test}} = S_{\text{val}} \cup S_{\text{test}}$. Note, in our setting, the random $D_{\text{test}}$ split to $S_{\text{val}} \cup S_{\text{test}}$ has been performed 15 times, to ensure that the set $S_{\text{test}}$ is not consistently the same. **Importantly, in both settings, there is no leakage of information from the test data, $S_{\text{test}}$ (used for performance evaluation) to the training of the model, $S_{\text{train}}$, and the calibration set of the detector, $S_{\text{val}}$.** We claim that these two partition strategies would give the same results. Let us elaborate. Given that both $D_{\text{train}}$ and $D_{\text{test}}$ are sampled i.i.d. from the same distribution $P$, **any partition strategy would produce sets, $S_{\text{train}}$, $S_{\text{val}}$ and $S_{\text{test}}$, which are all sampled i.i.d. from this same distribution $P$**. The only potential concern is the reduced sample size of the test set. In our experiments, the size of $S_{\text{test}}$ is $1,000$ -- the maximum window size. This choice was guided by our empirical findings that most metrics get a nearly perfect score around this size (for the top-performing methods). For example, our method exhibits a score of 99+ for the AUROC, AUPR-In, and AUPR-Out metrics for a window of size $1,000$, for all architectures considered; see Table 2 in the paper. Therefore, since a maximum size of $1,000$ samples seems sufficient for evaluating window-based detection methods, we argue that both settings would yield the same results. Finally, To further clarify our previous points regarding the references mentioned earlier, we have included those key citations: From reference [1]: "The noise magnitude $\epsilon$ was selected on a separate validation dataset, which is different from the out-of-distribution test sets". From reference [2]: "In all experiments, we first train a classifier $f_\theta$ on an in-distribution training set, then tune the hyperparameters (e.g. the perturbation magnitude $\epsilon$) on an in-distribution validation set without using its class labels...". From reference [25]: "we choose the weight of each layer $\alpha_l$ by training a logistic regression detector using validation samples." Thus, our use of a separate validation set is analogous to that in those references. Please let us know if this explanation makes sense to you. We remain available to answer any additional questions or offer further explanations. **References:** [1] - Enhancing the reliability of out-of-distribution image detection in neural networks. ICLR. [2] - Generalized ODIN: detecting out-of-distribution image without learning from out-of-distribution data. CVPR. [25] - A simple unified framework for detecting out-of-distribution samples and adversarial attacks. NeurIPS.
Let me first express my appreciation to the authors for their willingness to engage in this discussion. The way that you defined the problem in terms of $S_{train}, S_{test},$ and $S_{val}$ is precisely the point I was alluding to. Thank you for stating it in crisp and clear terms. Also, I do fully agree that there is no leakage from $S_{test}$ to $S_{train}$ or $S_{val}$. In fact, I stated so as much in one on my previous responses ("*I do understand that a separate hold-out validation set of 49000 samples has been created and that final results have not been reported on this held-out set*"). The concern was not that there was overlap between test set and val set samples, but rather that a phenomenon similar to the "generalization gap" would occur. This, as the authors are probably well familiar with, is the problem that accuracy on test-set is less than that on train-set even though train and test are sampled i.i.d from the same underlying distribution. This was more my primary concern, and I am still of the opinion that if the protocol followed was $D_{train} = S_{train} \cup S_{val}$ instead of $D_{test} = S_{test} \cup S_{val}$, then there would likely be a drop in performance. Having said that, since the authors mention having validated this across 15 different random splits of $D_{test}$ into $S_{test}$ and $S_{val}$, it does increase confidence in their reported outcomes. The authors are also right that the stated references have used validation sets to obtain values of hyper-parameters like $\epsilon$ and $\alpha_l$. This, I feel, is more in the spirit of hyper-parameter tuning and would be akin to the authors using a validation set to determine the optimal value of window-size $k$, but not to train the detection module itself. Nonetheless, the other methods are in fact using small held-out validation sets sampled from the test; so thanks for pointing this out.
Thanks for your detailed response. After further reading through [10] and some of the more recently published works which cite it there are clearly a range of further evaluation methods that have been proposed. Maggio & Dreyfus-Schmidt in their 2021 work "Ensembling Shift Detectors: an Extensive Empirical Evaluation" show synthetic shifts applied to 21 different classification tasks from OpenML and Kaggle. This seems like a more general evaluation than the currently proposed one which would improve the ability to evaluate the methodology. However, the methods proposed in this work are sound enough that I will keep my current rating without any additional empirical evaluation. As for the other questions, your responses are satisfying.
I thank the authors for all the effort put in the rebuttal and for the additional experiments. 1. I believe the two-sided version of the algorithm is an interesting theoretical contribution as well, even though less practical. Thus I suggest adding it to a section in the appendix as you have already studied it, and it might be interesting in the future with other types of drift or attacks. 2. Thank you for elaborating on this example to motivate Contribution (1). Having it early in the paper might help the reader better understand in which direction you are going. 3. The experiment with a mixed window of ID and OOD data is important, as it reflects better challenging application scenarios. I'm glad that you ran experiments with it. 4. Thank you for clarifying certain aspects w.r.t references [11,12]. 5. This might be just a question of writing style, but I suggest the authors to re-work the paragraph in which they segment their objective and the limitations into two separate paragraphs (L.26-41). I reckon it contains too much information. All of my concerns were addressed satisfactorily. With the richer experimental section and better motivation, I increase my recommendation for acceptance.
Thanks to the authors for providing a detailed rebuttal to all reviewers and engaging in the discussion. I read all the other reviewers' comments and the author's responses. I do not have any further concerns about the paper and support its acceptance. I suggest authors revise the paper considering the reviewers' comments.