PointAD: Comprehending 3D Anomalies from Points and Pixels for Zero-shot 3D Anomaly Detection

Summary

The paper titled "PointAD: Comprehending 3D Anomalies from Points and Pixels for Zero-shot 3D Anomaly Detection" introduces PointAD, a novel framework designed for zero-shot 3D anomaly detection by leveraging both point clouds and RGB images. The approach builds on the strong generalization capabilities of CLIP and adapts it to 3D anomaly detection. The framework renders 3D anomalies into multiple 2D views and then projects these 2D representations back into 3D space. This hybrid representation learning allows for capturing generic anomaly semantics across various unseen 3D objects. PointAD also incorporates auxiliary point clouds to optimize learnable text prompts for better anomaly detection and segmentation.

Strengths

1. The use of CLIP for 3D anomaly detection and the hybrid representation learning approach are novel contributions. 2. The theoretical foundations are robust, and the empirical validation is comprehensive. 3. The paper is clearly written, with well-structured sections and effective use of visual aids. 4. The approach addresses a critical gap in the field, with potential for significant impact on future research and practical applications.

Weaknesses

1. The sensitivity of the method to the selection of rendering angles and the number of views could be explored further. 2. While the experimental results are compelling, additional validation on more diverse and challenging datasets would further strengthen the claims. 3. The discussion of potential limitations and future work could be expanded to provide a more comprehensive view of the method's applicability and areas for improvement.

Questions

1. Could the authors provide more insights into the selection of rendering angles and the impact of different numbers of views on the detection performance? 2. Have the authors considered the robustness of the method under different environmental conditions, such as varying lighting and occlusions? 3. Could additional experiments on other industrial or medical datasets help to further validate the generalizability of the proposed method? ======== post rebuttal ========== The authors' rebuttal solve most of my concerns, hence I raise my score to borderline accept.

Rating

5

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors have addressed some limitations, including the fixed rendering angles and the potential for further optimization of view selection. Constructive suggestions for improvement include exploring the sensitivity to rendering parameters and validating the method under different environmental conditions.

Summary

The paper introduces PointAD, the first approach to explore the domain of zero-shot 3D anomaly detection, leveraging CLIP's strong generalization to identify anomalies in previously unseen objects. It offers a unified framework that integrates 3D point clouds with 2D renderings, employing hybrid representation learning to capture the semantics of anomalies. PointAD's key contributions are its pioneering exploration of the ZS 3D anomaly detection domain, its ability to seamlessly integrate multimodal data like RGB information for enhanced detection, and its superior performance over existing methods. The robustness of the framework is confirmed through extensive experiments.

Strengths

Originality: The paper introduces PointAD, a novel method in 3D anomaly detection that leverages CLIP for 3D analysis, uniquely combining point clouds and pixel data. It expands the application of vision-language models into new domains, showcasing versatility in 3D point cloud analysis. Soundness: The paper exhibits methodological soundness through its rigorous experimental setup, including the use of diverse datasets and a thorough ablation study that substantiates the design decisions and effectiveness of the proposed PointAD framework. The state-of-the-art experiments performance confirms the model's outperforming ability. Clarity: The paper stands out for its clarity, guiding readers smoothly from the problem statement to the final results. It skillfully explains complex concepts in an accessible way, ensuring that a wider audience can follow along. The paper also benefits from helpful visual aids that clearly demonstrate the model's effectiveness.

Weaknesses

1. There’s some zero-shot anomaly detection methods with clip on 2D images, so this paper should compare with them, especially, from the view of techniques’ differences, not only the used data differences. For example, those zero-shot methods (WinCLIP, VAND and AnomalyCLIP) mentioned in [R1] and [R2]. [R1] Cao, Y., Xu, X., Zhang, J., Cheng, Y., Huang, X., Pang, G., & Shen, W. (2024). A survey on visual anomaly detection: Challenge, approach, and prospect. arxiv preprint arxiv:2401.16402. [R2] Li, X., Huang, Z., Xue, F., & Zhou, Y. (2024). Musc: Zero-shot industrial anomaly classification and segmentation with mutual scoring of the unlabeled images. arxiv preprint arxiv:2401.16753. 2. I also wonder if the new methods of 3D anomaly detection can also be used to deal with this zero-shot task. For example, [R3] and [R4]. [R3] Zuo, Z., Dong, J., Wu, Y., Qu, Y., & Wu, Z. (2024). CLIP3D-AD: Extending CLIP for 3D Few-Shot Anomaly Detection with Multi-View Images Generation. arxiv preprint arxiv:2406.18941. [R4] Li, W., Xu, X., Gu, Y., Zheng, B., Gao, S., & Wu, Y. (2024). Towards Scalable 3D Anomaly Detection and Localization: A Benchmark via 3D Anomaly Synthesis and A Self-Supervised Learning Network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 22207-22216). 3. More importantly, there’s some very similar studies with this work and task. Authors should mention and compare them within this study. For example, [R5]. [R5] Li, Y., Goodge, A., Liu, F., & Foo, C. S. (2024). Promptad: Zero-shot anomaly detection using text prompts. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (pp. 1093-1102). [R6] X. Chen, J. Zhang, G. Tian, H. He, W. Zhang, Y. Wang, C. Wang,Y. Wu, and Y. Liu, “Clip-ad: A language-guided staged dual- path model for zero-shot anomaly detection,” arXiv preprint arXiv:2311.00453, 2023. [R7] Wang, C., Zhu, H., Peng, J., Wang, Y., Yi, R., Wu, Y., ... & Zhang, J. (2024). M3DM-NR: RGB-3D Noisy-Resistant Industrial Anomaly Detection via Multimodal Denoising. arxiv preprint arxiv:2406.02263. 4. Computational Efficiency: Although the authors have commendably addressed computational challenges, the overall computational cost associated with rendering multi-view images and processing them through the vision encoder is still substantial. Since this field is still developing with few comparable studies, it is recommended that the authors provide a broader comparison about the computational cost with existing anomaly detection techniques, including those like Cheraghian, CPFM and 3DSR that are not strictly within the zero-shot category. This expanded comparison will help to show that the computational demands of PointAD are reasonable and suitable for practical use cases. 5. Clarity and Completeness of Presentation: The paper's writing is generally clear and free of errors, yet minor aspects of the presentation could be polished for better reader comprehension. For instance, the figure overlays, specifically in Figure 1, currently only elucidate part (a), leaving explanations for parts (b) and (c) to be found in the main text. This separation can disrupt the reader's flow and understanding. Enhancing the figure overlays to include all relevant parts and ensuring that each component of the figure is self-explanatory will make the paper more accessible and its findings easier to grasp.

Questions

1. Please answer the main differences with these mentioned methods in Weaknesses and compare with part of them on experiments. 2. Given the concerns raised in the Weaknesses section 1, does PointAD still hold the merit of computational efficiency when compared to other methods such as AnomalyCLIP and Cheraghian? 3. Could you elaborate on the multi-view rendering process of point clouds (as described from line 134 to line 140), particularly how the color of the rendered images is determined? Furthermore, at line 254, it's stated that additional RGB information is utilized only in the test dataset. Given the description at line 564, which refers to 2D RGB information, how is this 2D data integrated with the rendered multi-view images? Are there any strategies which are employed to bridge the gap between the rendered training data and the RGB information used in testing?

Rating

6

Confidence

5

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors have discussed the limitations of their model, and there are no potential negative societal impacts associated with their work.

Summary

The paper propose a unified framework, PointAD,to detect 3D anomalies in a ZS manner. Hybrid representation learning is proposed to incorporate the generic normality and abnormality semantics into PointAD. PointAD can incorporate 2D RGB information in a plug-and-play manner for testing, which can perform ZS M3D anomaly detection directly. Experiments results demonstrate the superiority of the model in detecting and segmenting 3D anomalies.

Strengths

1. PointAD introduces a novel method for Zero-Shot 3D anomaly detection by integrating the strong generalization capabilities of CLIP to recognize 3D anomalies from unseen objects. This approach is first of its kind in ZS 3D anomaly detection. 2. The paper describes a unified framework that comprehends 3D anomalies from both points and pixels, allowing for a comprehensive understanding of anomalies by rendering 3D objects into 2D renderings and projecting them back to 3D space. It proposes hybrid representation learning to optimize learnable text prompts from both 3D and 2D data, enhancing the model's ability to generalize across diverse unseen objects. 3. Extensive experiments demonstrate the superiority of PointAD over existing state-of-the-art methods, especially in ZS settings where traditional models struggle.

Weaknesses

I do not notice any obvious weaknesses in this paper, but I still have a few questions that I hope the authors can answer. 1. The methodology, while innovative, is complex due to the need to render 3D objects into multiple 2D views and then re-project them, the performance of the model may heavily depends on the quality of the 2D renderings of the 3D objects, Therefore, if the renderings are of poor quality, could the model's performance degrade? 2. The experiments in the paper utilize high-resolution point cloud scans. Given that the point clouds obtained in practical applications may not always be high resolution, how effective is this method in detecting anomalies in low-resolution, sparse scans? 3. In the appendix's failure case section, the detection of the tire failed on the point score map. Thus, if a tire, or any object, has noise, how significantly does it affect the recognition on the point score map? I suspect that the possible reason for this failure could be the algorithm's sensitivity to point density and spatial distribution. When anomalies are located in areas where point density or distribution are inconsistent or significantly different from the training data, can this model still generalize well to new, unseen anomalies?

Questions

See weaknesses.

Rating

7

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors adequately addressd the limitations. Societal impact is not discussed, but I believe it's not needed.