Hierarchical Vector Quantized Transformer for Multi-class Unsupervised Anomaly Detection

Summary

This work builds a unified framework for multi-class anomaly detection (AD) by using normal images only. It identifies the identical shortcut issue of the reconstruction-based AD methods and tries to alleviate it by augmenting the memory with hierarchical discrete iconic prototypes. A switching mechanism is employed to deal with multi-class scenario. Furthermore, the hierarchical prototype-oriented optimal transport module is used to calibrate the anomaly scores. Experiments on MvTec-AD and CIFAR-10 demonstrate the superiority of the method.

Strengths

1. The motivation of using discrete iconic prototypes makes sense to alleviate the “shortcut” of reconstruction on abnormal images. 2. It achieves state-of-the-art performance on the two popular benchmarks. 3. The paper is easy-to-follow and well-written with clear motivation for each module.

Weaknesses

1. Missing necessary comparison on additional datasets, e.g., the VisA dataset [1], a multi-class industrial anomaly detection dataset. 2. As a core contribution, the details of vector quantization (VQ) (e.g., the quantization method) are not thoroughly presented and some related questions are not discussed, e.g., how different quantization precisions/methods affect the performance? [1] Zou, Yang, Jongheon Jeong, Latha Pemula, Dongqing Zhang, and Onkar Dabeer. "Spot-the-difference self-supervised pre-training for anomaly detection and segmentation." In European Conference on Computer Vision, 2022.

Questions

1. In Table 1, why the results under one-for-all setting are better than that under one-for-one setting? For example, the results on Capsule is far lower under one-for-one setting. Though the authors claim that the increased data diversity is beneficial, it requires more experiments to support the argument, e.g., try taking images from different N (N >= 2) classes for training. 2. The ablation study is incomplete. Missing the results when only Hierarchical VQ is enabled.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

Limitations are discussed in the Discussion section and potential negative societal impacts are not mentioned.

Summary

This paper proposes a variational autoencoding framework for unsupervised anomaly detection. This work addresses the identical shortcut issue by preserving the typical normal patterns as discrete iconic prototypes and also overcomes the problems of prototype collapse problem.

Strengths

It designs a network from the perspective of solving an issue identical shortcut. The method achieves good experimental results on classical datasets.

Weaknesses

1. The descriptions of the figures need to be clearer. For example, in Fig.1, it would be helpful to label each image and coordinate system to indicate what they represent. Additionally, labeling the subfigures with (a) and (b) would make the explanation clearer, such as in Fig.5. 2. The performance on chosen datasets are close to saturation, especially in some categories where the results achieve 100, this is unreasonable. In this field, more challenging datasets should be tested to further push the boundaries of research. 3. The performance of proposed method under one-for-one setting still has a gap compared to the SOTA method. 4. The innovation is limited, where most of the ideas have already be proposed.

Questions

Why the proposed methods under one-for-one setting has lower performance than the one-for-all setting, which differs from those in most other methods? What is the reason for the significant performance differences of the proposed method on different categories, such as poor performance on “Toothbrush”, and is this explainable?

Rating

3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

1 poor

Presentation

2 fair

Contribution

1 poor

Limitations

The authors have clarified the limitation of this method that the category labels are assumed to be available during the training stage， and identify this issue as future study by incorporating the model with clustering methods.

Summary

This paper proposes a feature reconstruction based framework for multi-class anomaly detection, called hierarchical vector quantized Transformer (HVQ-Trans). To address the "identical shortcut" problem occurring in the reconstruction-based framework, the proposed method replaces the original encoding features with the nearest iconic prototypes learned from normal training data, and then decoded with a VQ-based transformer decoder to reconstruct the anomaly regions into normal regions. Besides, a hierarchical prototype-oriented learning and anomaly scoring method is developed to guide prototype learning and accurately identify anomalies.

Strengths

1. The motivation is clear and reasonable 2. The writing quality and paper structure are good. 3. The idea of using vector quantization for feature reconstruction in anomaly detection is interesting. The proposed method is technically sound. 4. The proposed method obtains decent performance improvement than the current methods.

Weaknesses

1. Using prototypes may lose high-frequency information leading to imprecise feature reconstruction. 2. Is the optimal transport necessary? How about only simply using similarity scores as the weights? 3. The proposed HVQ-Trans has large model parameters. Please compare the proposed method with UniAD with the inference speed and model parameters. 4. Please add more visualization comparisons with related methods, such as UniAD.

Questions

see the weaknesses

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

see the weaknesses

Summary

This paper introduces a novel approach to multi-class anomaly detection (AD) by integrating hierarchical embedding vector quantization. To tackle the problem of identical shortcuts in the reconstruction-based AD paradigm, the authors suggest to enlarge abnormality's reconstruction residue by introducing discrete prototypes in the model. Additionally, the model incorporates a switching mechanism to further improve the feature reconstruction process. The proposed model is evaluated on MVTec and CIFAR-10 datasets and compared against prior arts.

Strengths

1. The paper is well-written and easy to follow. 2. The hierarchical vector quantized transformer is well presented. Appendix about the loss back-propagation explains parameter updates regardless quantization operation. 3. Extensive experiments are conducted. Ablation studies on model components suggest effectiveness of the proposed method.

Weaknesses

1. The vector quantization mechanism described in this paper appears to be a specific instance of the memory mechanism, where a continuous-valued feature is substituted with a numerical prototype. If this is indeed the case, such a memory mechanism has been widely employed in previous studies on AD. 2. What does 'C' represent in Figure 2 and Figure 3? Is it indicating feature concatenation? The paper briefly mentions feature aggregation but does not provide a specific definition or explanation of the feature aggregation operation. 3. Within the switching mechanism, a multi-category classifier, N codebooks, and reconstruction experts are necessary. In this scenario, can we consider the model as a combination of a single encoder and N decoders that are specific to each class? 4. The experimental evaluation compares the proposed method with US, PSVDD, PaDim, CuPaste, MKD, and DREAM. However, there exist several recently proposed AD models, such as Patchcore [1], RD4AD [2], CS-Flow [3], UTRAD [4], which outperform the aforementioned methods. It is highly recommended to include these methods in the experimental analysis as well. [1] Karsten Roth et al. “Towards total recall in industrial anomaly detection”. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022, pp. 14318– 14328. [2] Hanqiu Deng and Xingyu Li. “Anomaly Detection via Reverse Distillation from One-Class Embedding”. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022, pp. 9737–9746. [3] Marco Rudolph et al. “Fully convolutional cross-scale-flows for image-based defect detection”. Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 2022, pp. 1088–1097. [4] Liyang Chen et al. “UTRAD: Anomaly detection and localization with U-Transformer”. Neural Networks 147 (2022), pp. 53–62.

Questions

Please refer to the weakness section. Thanks.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

No discussion on limitations and potential negative social impact.

Summary

The paper proposes using hierarchical vector-quantized transformer-based autoencoders for image anomaly detection. The key contribution is the use of prototypes learned using an optimal transport algorithm.

Strengths

The paper addresses a critical problem in reconstruction-based anomaly detection (good reconstruction of both normal samples and anomalies) and proposes a reasonably solid solution based on discrete latent prototypes.

Weaknesses

1) The first and foremost problem with the proposed approach is the claim that it is "unsupervised". As the paper admits in the final discussion paragraph on page 9, the current work assumes that the category labels are available during the training stage. In that case, how can it be claimed that the proposed method is unsupervised? Also, this may make comparison with unsupervised approaches such as UniAD unfair. 2) The second critical issue is the novelty of the proposed solution because it combines some well-known ideas such as vector-quantized VAE [21] and hierarchical VQ-VAE [17] and some recent ideas such as learning latent prototypes for autoencoders using optimal transport [A, B]. [A] Bie et al., "Learning Discrete Representation with Optimal Transport Quantized Autoencoders", 2023 [B] Oliveira et al., "Improving Variational Autoencoders Reconstruction Using Prototypes", 2023. However, in my opinion, the paper smartly combines these known ideas and makes some marginal improvements to implement a transformer-based VAE model. 3) The switching mechanism and mixture-of-experts concepts on page 4 have not been clearly explained. In particular, it is not clear what the so-called "classifier for producing logits" and "expert" mean and how these components tie in with the POT module described subsequently. Furthermore, the ablation study talks about "codebook switching" and "expert switching", but these are never described in the paper. 4) Though the paper claims that the proposed method achieves SOTA results, it is not obvious if this is true. Except for UniAD (published in NeurIPS 2002), all the other baseline methods selected for benchmarking are from 2020 or 2021. A cursory glance at more recent works such as [C, D] indicates that the reported performance results fall short of SOTA results. [C] Liu et al., "SimpleNet: A Simple Network for Image Anomaly Detection and Localization" CVPR 2023 [D] Tien et al., "Revisiting Reverse Distillation for Anomaly Detection", CVPR 2023 5) The calibration of anomaly score based on POT has been emphasized many times, but there appears to be no experiment to demonstrates the importance of this idea. 6) It is not clear how to determine the number of prototypes required? Will this depend on the number of classes (or number of anomaly types)? Why does the performance start decreasing when the number of prototypes start increasing beyond 512? 7) There is no mention about the computational complexity of the proposed approach.

Questions

Please see weaknesses.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

All the limitations of the proposed method have not been discussed and addressed. The paper does not appear to have any potential negative social impact.

Response to Author Rebuttal

I thank the authors for their detailed response, which address some of my core concerns (regarding the switching mechanism and unsupervised claim). Hence, I'm inclined to increase my rating.

Thanks for the feedback from authors.

The authors answer most of my questions. But I am still concerning the innovation of the vector quantization mechanism. So I keep my original rating.

I keep my initial decision!

After carefully read the rebuttal, I think this paper which with the problems of technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations, are still unsolved.

The decision is keeping consistent.

The method is a stacked work with the previous techniques, the experimental results and analysis are not attracting.

Keep my rating

I appreciate that the authors provide additional experiments (main results on the VisA dataset and the ablation study) and detailed descriptions on the vector quantization. After carefully investigating other reviewers' comments and the rebuttal, the authors solve most of my concerns; however, I would like to keep my original rating due to the limited technical novelty of quantization on memory mechanism, which has been widely explored in the literature.