Summary
This paper presents a novel approach to Out-Of-Distribution (OOD) detection in the realm of 3D data, termed OpenPatch. It addresses the challenge of detecting novel objects (OOD data) that a deep learning model has not encountered during its training phase. The core contributions of the paper can be summarized as follows:
(1) Development of OpenPatch: OpenPatch, a novel OOD detection method, leverages patch representations from intermediate layers of pre-trained 3D models. This approach enables the detection of novel 3D objects by differentiating between known and unknown object categories without the need for any additional training, addressing the limitations of previous methods that rely heavily on fine-tuning with extensive support data sets.
(2) Plug-and-Play Capability with Semantic Novelty Detection: OpenPatch operates as a plug-and-play solution for real-world applications, notably in industrial robotics, where computational resources and available data are limited. It is designed to detect semantic novelty effectively by comparing the test sample’s patch features against a database of known classes, using distance metrics and entropy-based class assignment diversity to identify OOD instances, thereby mitigating potential operational hazards.
(3) Demonstrated Superiority and Efficiency: The approach not only surpasses existing OOD detection methods in performance but also showcases high sample efficiency and resilience to domain bias. This makes OpenPatch highly suitable for practical applications, as it can be readily deployed without retraining for different tasks or updating the nominal support set, which is a significant advancement over existing 2D and 3D OOD detection techniques.
In summary, the paper introduces a robust and efficient method for detecting new objects in 3D data, which is of particular relevance for industrial applications and other real-world scenarios where data and computational resources are constrained.
Strengths
OpenPatch's strengths are concentrated in the following areas:
(1) Novelty in distinguishing known and unknown classes: OpenPatch introduces a strategy for extracting generalizable patch features from pre-trained 3D deep learning architectures. It devises an innovative approach that integrates semantic and relative distance information to accurately identify new categories during the testing phase.
(2) Streamlined deployment: OpenPatch can be efficiently deployed in resource-constrained environments, obviating the need for collecting custom datasets and training task-specific models.
Weaknesses
(1) About innovation.
The approach taken in this paper is too similar to PatchCore. For example, the "Patch Feature Extractor" in OpenPatch mirrors the "Local-Aware Patch Feature" in PatchCore, while the "Memory Bank and Subsampling" in OpenPatch is very similar to the "Coreset Reduced Patch Feature Memory Bank" in PatchCore. And the key strategies selected by the two papers are similar, such as Greedy Coreset and KNN. Overall, I think this paper lacks innovation.
(2) About experiments.
A. The selected comparison methods are out of date. For example, in Tables 1 and 2 the chosen comparison methods are EVM Rudd et al. (2015) and Mahalanobis Lee et al. (2018), have become obsolete.
B. The experimental advantages are not obvious. For example, in the "Training on Support Set" table, only 3 out of 6 main metrics outperform previous methods.
(3) About writing.
The paper requires improvement in both writing and logic, particularly in the abstract and introduction sections.
Questions
(1) About experiments.
Table 1 and Table 2 demonstrate the performance advantages of OpenPatch on OOD compared with other methods. Moreover, the experiments in Table1 are pre-trained on Objaverse-LVIS dataset, while Table 2 shows the results obtained when starting from the OpenShape pre-trained multimodal embedding. But I have some questions about the choices of comparison methods:
A.In Table 1 and Table 2, compared with methods such as EVM and Mahalanobis that have been proposed for many years, if the latest research such as [1][2][3] are added to the comparison, can OpenPatch maintain its advantage?
[1]Semantic Novelty Detection via Relational Reasoning
[2]Detecting out-of-distribution examples with Gram matrices
[3]Delving into Out-of-Distribution Detection with Vision-Language Representations
B.If the latest research [4] and [5] are introduced into the backbone selection of Table 2, what changes will occur in the experimental results?
[4]Uni3D: Exploring Unified 3D Representation at Scale
[5]ViT-Lens: Towards Omni-modal Representations
(2) As mentioned in 3.2, "The banks" cardinality may quickly increase, significantly impacting the computational cost of the method. To mitigate this effect and address redundancy we adopt a greedy coreset subselection mechanism.” Can the impact of greedy coreset subselection mechanism on computational cost reduction be quantitatively reflected in the experiment?
Rating
3: reject, not good enough
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.