MultiOOD: Scaling Out-of-Distribution Detection for Multiple Modalities

Summary

This paper first proposes a MultiOOD benchmark for multimodal OOD detection. It then further proposes methods including A2D and NP-Mix for better tacking the multimodal OOD detection task.

Strengths

1. I appreciate the formulation of the multimodal OOD detection benchmark. 2. This work presents extensive experiments and achieves good results.

Weaknesses

(See the questions for more details.)

Questions

Overall, I am now hesitating around the borderline for this paper and below are my concerns: 1. Generally, I believe that the relationship between this work and other multi-modality works can be very close. I thus suggest the authors to also discuss with other multi-modality works in their related works. I will also elaborate on this more in below concerns. 2. As for the benchmark, I believe that it is important for the evaluation to be conducted in a more statistically significant manner. In other words, from my perspective, the authors may consider to at least evaluate methods on different ID/OOD category splits and report standard deviation. 3. W.r.t. A2D in Sec 4.2, from my perspective, it seems that sometimes, the orders of non-ground-truth classes matter. For example, while for a cat input, both fox and truck are non-ground-truth classes, it seems natural for fox to have a higher softmax score than truck. Thus, I am curious, what if the discrepancy is maximized in an order-preserved manner. I appreciate if this can be performed as an ablation study to see whether this alternative of A2D can be more effective. 4. W.r.t. Sec 4.3, the authors claim that they get inspiration from [16]. I thus hope to see a more detailed comparison between the proposed method and the method in [16]. Besides, it seems to also worth comparing with Learning Placeholders for Open-Set Recognition CVPR 2021. If I am not wrong, it seems that the proposed NP-Mix is somehow similar to its data placeholder. 5. As for the evaluation, as mentioned in the first concern, this work is closedly related to those multi-modaity ones. Thus, to better validate the efficacy of A2D and NP-Mix, besides comparing with only uni-modal OOD methods, it is also suggeted to create baselines that combine uni-modal OOD methods and existing typical multi-modality methods outside the OOD area. This can lead the proposed method to be better evaluated.

Rating

6

Confidence

3

Soundness

2

Presentation

3

Contribution

2

Limitations

This work has discussed its potential limitations.

Summary

This paper introduces MultiOOD, a new benchmark for multimodal out-of-distribution (OOD) detection. The authors propose two new techniques: 1) Agree-to-Disagree (A2D), which encourages discrepancy between modality predictions during training, and 2) NP-Mix, a novel outlier synthesis method. Extensive experiments on the MultiOOD benchmark demonstrate significant improvements over existing unimodal OOD detection methods.

Strengths

Originality: -First benchmark for multimodal OOD detection (MultiOOD). -A2D algorithm leveraging modality prediction discrepancy. -Outlier synthesis method (NP-Mix). Quality: -Strong performance improvements over baselines. -Thoughtful analysis of modality prediction discrepancy phenomenon. -Comprehensive experiments on diverse datasets. Clarity: -Well-organized structure. -Clear explanations of key concepts and methods.

Weaknesses

Weaknesses: 1) Lacks important baselines (e.g. ensemble of multiple singleOOD methods for each modality). This also relates to the question of "why we should study MultiOOD?" The importance of studying MultiOOD and its practical impact on real-world applications could be further demonstrated. 2) Limited theoretical analysis of proposed methods. A2D and NP-Mix need more theoretical analysis. 3) The experiments are only conducted on the task of action recognition. In this case, the title "MultiOOD" seems over-claiming. Please specify the task in the title and the main text. In addition, it only focuses on specific modalities (video, optical flow, audio). This makes the scope of the paper somewhat limited. 5) Some figures (e.g. Fig. 4) could be improved for clarity, with consistent color-coding and same x-axis scales.

Questions

1) Why "α > 1" can ensure the synthesized outliers reside in the intermediary space between two prototypes, rather than near the ID data? Please give more explanations. 2) How sensitive are the A2D and NP-Mix methods to hyperparameter choices?

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors acknowledge some limitations, such as the focus on specific modalities (video, optical flow, audio) and action recognition tasks. However, they could further discuss potential limitations in scaling to a larger number of modalities or very different types of data.

Summary

The paper introduces a novel OOD detection benchmark for multimodal data (called MultiOOD), covering diverse dataset sizes and modalities. Based on this benchmark, the authors first demonstrate the Modality Prediction Discrepancy phenomenon, which means that the discrepancies of softmax predictions are shown to be negligible for in-distribution data (across different modalities) and significant for OOD data. Based on these observations, the authors introduce a novel Agree-to-Disagree (A2D) algorithm, which aims to enhance such discrepancies during training. Additionally, the authors propose a new outlier synthesis algorithm NP-Mix that explores broader feature spaces and complements A2D to strengthen the OOD detection performance. Experimental validation is extensive and confirms the improvements in OOD detection due to the new proposed algorithms.

Strengths

Paper's strengths: - the paper has a significant scientific contribution, by introducing the first multi-modal benchmark for OOD detection - besides this, it contributes to the improvement of OOD detection by introducing two new algorithms: A2D algorithm and a new outlier synthesis algorithm NP-Mix that explores broader feature spaces and complements A2D - the paper is clearly written and well-documented - the review of the state of the art is comprehensive and covers most of the relevant work - experimental validation is extensive and underline the improvements introduced by the A2D and NP-Mix algorithms for OOD detection

Weaknesses

- Some more details are required regarding some aspects.

Questions

Here are my concerns: - In the video modality, what features do you extract to characterize the stream? Do you extract per-frame features, or per-video features. - Same question for the audio modality - You use 5 datasets, all of them having the video and optical flow modalities, but audio modality is missing from 2 of them. In this case, what protocol do you adopt? Do you ignore them and perform the audio analysis only on the remaining three? - Is your approach robust when one modality is missing?

Rating

8

Confidence

4

Soundness

4

Presentation

4

Contribution

4

Limitations

The identified limitations are: (i) the performance on Near-OOD benchmark with a large number of classes can be further improved; and (ii) the ID/OOD discrepancy could also be further improved. The paper does not represent any negative societal impact.

Thank you for the rebuttal.

Thanks for the rebuttal. Most of my concerns are well-solved. The reviewer suggests adding these information to the revision to make the paper stronger. I will increase my rating to weak accept.

Acknowledgement of Rebuttal

I want to thank the authors for addressing all my concerns.