Recent advancements in anomaly detection have seen the efficacy of CNN- and transformer-based approaches. However, CNNs struggle with long-range dependencies, while transformers are burdened by quadratic computational complexity. Mamba-based models, with their superior long-range modeling and linear efficiency, have garnered substantial attention. This study pioneers the application of Mamba to multi-class unsupervised anomaly detection, presenting MambaAD, which consists of a pre-trained encoder and a Mamba decoder featuring (Locality-Enhanced State Space) LSS modules at multi-scales. The proposed LSS module, integrating parallel cascaded (Hybrid State Space) HSS blocks and multi-kernel convolutions operations, effectively captures both long-range and local information. The HSS block, utilizing (Hybrid Scanning) HS encoders, encodes feature maps into five scanning methods and eight directions, thereby strengthening global connections through the (State Space Model) SSM. The use of Hilbert scanning and eight directions significantly improves feature sequence modeling. Comprehensive experiments on six diverse anomaly detection datasets and seven metrics demonstrate state-of-the-art performance, substantiating the method's effectiveness. The code and models are available at https://lewandofskee.github.io/projects/MambaAD.
Paper
Similar papers
Peer review
Summary
The paper pioneers the utilization of Mamba in the anomaly detection field. Specifically, the core contribution of the proposed MambaAD is a Locality-Enhanced State Space module, which utilizes Mamba for global information and CNN for local information. Whereas the motivation and the core technical contribution of this paper are similar to other works like U-Mamba, VMamba, and MedMamba, the proposed MambaAD is the first to extensively explore Mamba in the anomaly detection field and achieve state-of-the-art multi-class anomaly detection performance with a low parameter count and computational demand, making this paper acceptable for me.
Strengths
This paper is the first to explore Mamba in the anomaly detection field. The proposed MambaAD achieves state-of-the-art multi-class anomaly detection performance with lower computational demand and fewer parameters than other multi-class anomaly detection alternatives. The experiments are extensive, featuring six distinct AD datasets with seven metrics.
Weaknesses
1. The motivation for using Mamba for visual anomaly detection should be strengthened. e.g., what’s the core challenge for multi-class anomaly detection? why long-range modeling is important for multi-class anomaly detection? 2. It would be better to report the performance with only global/local branches in Table 5, which can better show the influence of individual branches. 3. While the proposed MambaAD achieves better detection performance in most metrics on the evaluated datasets, for some metrics, like AUPRO on VisA, the proposed MambaAD is not the best one. It would be better to offer some analysis on this issue. 4. The current MambaAD framework appears to still rely on a pre-trained CNN encoder, utilizing the Mamba model only in the decoder. Why is this design chosen? What would be the implications if the entire structure were based on the Mamba model? 5. It seems that only the Hilbert scanning method is ultimately used out of the five proposed hybrid scanning methods. How should we further interpret the ablation study in Table 3?
Questions
See the weakness.
Rating
8
Confidence
5
Soundness
3
Presentation
4
Contribution
3
Limitations
See the weakness.
Summary
This paper employs a pyramid-structured auto-encoder to reconstruct multi-scale features, utilizing a pre-trained encoder and a decoder based on the Mamba architecture. The experimental results show SoTA performances on several commonly used datasets.
Strengths
1. As far as I am concerned, MambaAD is the first attempt to use Mamba in the anomaly detection domain. 2. The experimental results are SOTA. 3. The representation is clear and easy to follow.
Weaknesses
1. My major concern is the efficiency comparison shown in Table 4. UniAD is a transformer-based method, but leading to the smallest params and FLops. The advantage of Mamba should be the efficiency, however, which is not as good as transformer-baed methods. 2. The backbone model is WideResNet50. The backbone model of the other method is not WideResNet50. For example, UNIAD use the efficient-net. I am not sure about wether the improvements in this model imply in different backbone model.
Questions
1. Why the params and FLOPs are higher than the transformer-based model? 2. Why the evaluation metrics on the pixel level is lower? Is the anomaly localization worse?
Rating
5
Confidence
4
Soundness
3
Presentation
3
Contribution
3
Limitations
The limitation and broader societal impacts are mentioned in the last paragraph.
Summary
The paper "MambaAD: Exploring State Space Models for Multi-class Unsupervised Anomaly Detection" introduces MambaAD, a novel framework for multi-class unsupervised anomaly detection using Mamba-based models. The framework consists of a pre-trained encoder and a Mamba decoder that integrates Locality-Enhanced State Space (LSS) modules at multiple scales. The LSS module combines Hybrid State Space (HSS) blocks and multi-kernel convolution operations to capture both long-range and local information. The proposed method is evaluated on six diverse anomaly detection datasets, demonstrating state-of-the-art performance and significant improvements in efficiency and accuracy.
Strengths
1. The use of Mamba-based models for multi-class anomaly detection and the innovative LSS module are novel contributions. 2. The theoretical foundations are robust, and the empirical validation is comprehensive. 3. The paper is clearly written, with well-structured sections and effective use of visual aids. 4. The approach addresses a critical gap in anomaly detection research and demonstrates significant performance improvements, making it highly relevant and impactful.
Weaknesses
1. The sensitivity of the method to the selection of scanning methods and directions could be explored further. 2. While the experimental results are compelling, additional validation on more diverse and challenging datasets would further strengthen the claims. 3. The discussion of potential limitations and future work could be expanded to provide a more comprehensive view of the method's applicability and areas for improvement.
Questions
1. Could the authors provide more insights into the selection of scanning methods and the impact of different numbers of directions on the detection performance? 2. Have the authors considered the robustness of the method under different environmental conditions, such as varying lighting and occlusions? 3. Could additional experiments on other industrial or medical datasets help to further validate the generalizability of the proposed method? ===== post rebuttal ====== The authors' rebuttal solve most of my concerns, hence I raise my score to 7.
Rating
7
Confidence
3
Soundness
3
Presentation
3
Contribution
3
Limitations
The authors have adequately addressed some limitations, including potential issues with scanning method selection and environmental conditions. Constructive suggestions for improvement include exploring the sensitivity to scanning parameters and validating the method under different environmental conditions.
Thanks.
Dear reviewer aMDx: Thank you for recognizing our work. We commit to incorporating the content you suggested in the revised version. Thank you again for your effort in the review and the discussion! Best regards! Authors of MambaAD.
Summary
This paper introduces a method for multi-class unsupervised anomaly detection utilizing a CNN-based encoder and a Mamba-based decoder. It incorporates an LSS module and a HSS block within the Mamba decoder to enhance performance. The effectiveness of the method is validated through experiments conducted on three image anomaly detection datasets.
Strengths
1. This study represents a novel exploration in using Mamba for anomaly detection. As far as I understand, this is the first research endeavor to apply Mamba in the context of anomaly detection. 2. Extensive experiments are presented in the paper.
Weaknesses
1. From a methodological standpoint, the anomaly detection approach essentially implements RD4AD, with the CNN decoder replaced by a Mamba decoder. 2. Several key ablations are absent from the experimentation: (a) It would be nice to conduct a comparison study using the same RD4AD architecture with CNN-based decoder, transformer-based decoder, and Mamba-based decoder. This comparison would validate the effectiveness of Mamba compared to other decoder architectures. (b) Ablations of model components should be included, which compares the proposed method, the proposed method without the HSS block, and the proposed method without the LSS module. This ablation study would provide insights into the functionality of these components.
Questions
Please refer to the comments in the Weakness Section. --------------- Post-rebuttal review: While I still have some reservations about the methodological novelty of this study, the performance and generalizability of the method appear to be strong. After reviewing the authors' response and considering the feedback from other reviewers, I have decided to raise my initial rating.
Rating
5
Confidence
5
Soundness
2
Presentation
2
Contribution
1
Limitations
Minimal discussion. Although the section labeled "limitations" includes some wording, it does not actually discuss any perspectives on real limitations on the methods.
Response
After considering the author's responses to all reviewers during the rebuttal phase, I believe that the contribution and quality of this paper fully meet the high standards of NeurIPS. I strongly recommend accepting this paper and will modify my rate from 6 to 8. Here are my reasons: The authors are the first to apply Mamba to the field of anomaly detection, which represents a significant innovation. The introduction of the LSS and HSS modules demonstrates unique approaches and technical advantages in handling multi-class anomaly detection tasks. The experimental section is very detailed and convincing. The rebuttal provided extensive theoretical and experimental analyses that addressed most of my concerns. However, there is still some room for minor improvements. I suggest including the comparative experimental analysis with Transformer-based methods in the main text. Additionally, incorporating the pixel-level evaluation metrics in the main text would further substantiate the effectiveness of the approach in anomaly localization. Also, it will be interesting to study the influence of different input resolutions, since Mamba should have better long-term information acquisition capability.
Thanks and response to resolutions.
Dear Reviewer wfnx, **Thank you for recognizing our work and your valuable suggestions to help us improve the manuscript.** We commit to *including ablation experiments based on Transformer methods corresponding to Tab. 1, 7, and 8 in the revised version*. Additionally, we will incorporate the *mIoU evaluation metric results across six datasets to further validate the effectiveness of our approach in anomaly localization*. **Q: Different Input Resolutions** ***(1)*** Regarding the impact of different resolutions, particularly on long-term information acquisition capability, we compared MambaAD's performance on the **MVTec-AD and VisA datasets at $256^2$ and $512^2$ resolutions**, as shown in the table below. For the MVTec-AD and VisA datasets, the **RD4AD method shows a significant decline in all metrics as the resolution increases**, indicating that *purely convolutional approaches struggle with long-distance modeling*. ***(2)*** For the Transformer-based UniAD method, increasing the resolution on the MVTec-AD dataset results in a **decline in image-level metrics, while pixel-level metrics such as P-AP, P-F1_max, P-AUPRO, and P-mIoU improve**. A similar trend is observed in MambaAD. This outcome may be due to the enhanced and clearer detailed information in higher resolutions, which **aids the model in more accurately identifying and distinguishing different regions for pixel-level tasks**, thereby improving anomaly localization metrics. However, for image-level classification tasks, **excessive detail may introduce noise and increase model complexity, leading to a decline in performance**. The decrease in the P-AUROC metric could be attributed to AUROC's nature as a measure of a classifier's overall performance across different thresholds, *reflecting the model's ability to distinguish between positive and negative samples*. *Higher resolution might cause instability in the model's performance at certain thresholds, especially for borderline samples, leading to a decrease in AUROC*. Overall, on the MVTec-AD dataset, the **MambaAD model outperforms both the CNN-based RD4AD and the Transformer-based UniAD at $512^2$ resolution**. ***(3)*** On the VisA dataset, the **UniAD method shows a general decline in performance at $512^2$ resolution compared to $256^2$ resolution**. This could be due to the *smaller anomaly areas in the VisA dataset, where Transformers may introduce noise during high-resolution global modeling and lack local region modeling capability*. For the **MambaAD** method, **increasing the resolution significantly improves all metrics except P-AUROC**. This improvement is attributed to the **effective long-distance modeling capability of Mamba and the integration of the proposed Locality-Enhanced LSS module. *The decline in P-AUROC might be due to changes in the distribution of positive and negative samples caused by the increased detail, affecting AUROC*. Overall, on the **VisA** dataset, **increasing the resolution significantly enhances MambaAD's performance compared to CNN and Transformer-based models**. In the future, we will continue to explore the impact of different resolutions, particularly higher resolutions such as $1024^2$ and $2048^2$, to demonstrate the effectiveness of the Mamba model in high-resolution anomaly detection. ***MVTec-AD Dataset*** | Resolution | Method | I-AUROC | I-AP | I-F1_max | P-AUROC | P-AP | P-F1_max | P-AUPRO | P-mIoU | | ---------- | ------- | ------ | ---- | ------ | ------ | ---- | ------ | ------ | ---- | | $256^2$ | RD4AD | 94.6 | 96.5 | 95.2 | 96.1 | 48.6 | 53.8 | 91.1 | 37.0 | | $256^2$ | UniAD | 96.5 | 98.8 | 96.2 | 96.8 | 43.4 | 49.5 | 90.7 | 32.5 | | $256^2$ | MambaAD | 98.6 | 99.6 | 97.8 | 97.7 | 56.3 | 59.2 | 93.1 | 41.2 | |$512^2$| RD4AD | 86.0 | 91.9 | 90.3 | 92.9 | 45.4 | 49.1 | 88.2 | 33.5 | |$512^2$ | UniAD | 96.3 | 98.5 | 95.5 | 95.6 | 46.9 | 50.1 | 91.5 | 33.7 | | $512^2$ | MambaAD | 97.7 | 99.0 | 96.3 | 96.8 | 60.4 | 60.9 | 93.1 | 44.5 | ***VisA Dataset*** | Resolution | Method | I-AUROC | I-AP | I-F1_max | P-AUROC | P-AP | P-F1_max | P-AUPRO | P-mIoU | | ---------- | ------- | ------ | ---- | ------ | ------ | ---- | ------ | ------ | ---- | | $256^2$ | RD4AD | 92.4 | 92.4 | 89.6 | 98.1 | 38.0 | 42.6 | 91.8 | 27.9 | |$256^2$ | UniAD | 88.8 | 90.8 | 85.8 | 98.3 | 33.7 | 39.0 | 85.5 | 25.7 | | $256^2$| MambaAD | 94.3 | 94.5 | 89.4 | 98.5 | 39.4 | 44.0 | 91.0 | 29.5 | | $512^2$| RD4AD | 89.1 | 90.2 | 86.9 | 93.8 | 29.8 | 37.0 | 89.7 | 23.2 | | $512^2$| UniAD | 87.4 | 88.5 | 84.8 | 96.6 | 26.7 | 35.2 | 87.2 | 22.0 | | $512^2$| MambaAD | 95.7 | 97.0 | 92.5 | 96.1 | 40.1 | 46.1 | 92.0 | 31.0 | Best regards! Authors of MambaAD.
Thanks for your detailed response. Some of my issues have been addressed. But why does the MambaAD with the backbone ResNet-34 and WideResNet-50 result in more parameters and Flops (compared to UNIAD)? Moreover, could you please list the inference time or FPS to show the inference speed of MambaAD and UniAD?
Response to Reviewer gwhg about FLOPs and FPS
Dear Reviewer gwhg: Thank you for your comments. **Q: FLOPs** (1) As shown in Fig. 1 of the main text, UniAD is a single-scale method, whereas our Mamba-based MambaAD is a multi-scale method. UniAD **concatenates features from the four stages of the encoder** and uses a Transformer to model **only the smallest scale feature map**. In contrast, MambaAD gradually **increases the resolution of the feature maps** during the decoder stage and performs modeling and reconstruction at four different scales. This difference in framework leads to MambaAD consuming more parameters and FLOPs **on high-resolution feature maps**. Additionally, the WideResNet-50 backbone network, with its higher number of channels at each scale, further increases the parameter and FLOP differences between the two methods. (2) The Mamba-based model offers **better interpretability and performance**. The model's FLOPs are also related to the implementation method. For instance, Mamba's **FLOPs could be reduced if the state transition is implemented using a vanilla for-loop**. However, the authors of Mamba chose to double the FLOPs to achieve a more parallel and efficient implementation. **Q: FPS** We evaluated the FPS of UniAD and MambaAD on ResNet-34 and WideResNet-50 backbones to demonstrate inference speed. The results show that MambaAD still lags behind UniAD in terms of inference speed. The following are possible reasons and directions for optimization: (1) Model framework: As mentioned in ***Q: FLOPs***, the MambaAD model requires more FLOPs, which affects inference speed to some extent. (2) Code implementation: The MambaAD code currently uses the initial Mamba implementation with mamba_ssm and causal-conv1d for SSM, which suffers from suboptimal algorithmic efficiency. Using the latest CUDA-accelerated selective_scan_cuda_core algorithm could improve computational efficiency by **40%**. (3) Number of scanning directions: The current method uses eight different scanning directions to ensure optimal performance. Using only **two different scanning directions** on the ResNet-34 backbone can still achieve an **mAD of 85.8 and an FPS of 574.6**, representing a **258% increase in FPS** compared to using eight scanning directions. (4) Code implementation of scanning methods: The current Hybrid Scan method is implemented solely in PyTorch, resulting in low efficiency. Our observations indicate that the HS encoder and HS decoder **occupy more than 60% of the time**. We plan to use **Triton or CUDA to accelerate** this process in the future. In summary, while MambaAD shows **significant performance improvements** over UniAD, its inference speed is affected by the aforementioned issues. We will continue to optimize the model and algorithms to surpass the inference speed of Transformer-based frameworks. | UniAD | Params(M) | FLOPs(G) | FPS | mAD | | ------------ | --------- | ----- | ---- | ---- | | ResNet-34 | 16.1 | 6.1 | 1119.9 | 78.2 | | WideResNet-50 | 33.5 | 14.4 | 582.7 | 78.9 | | MambaAD | Params(M) | FLOPs(G) | FPS | mAD | | ----------- | --------- | ------ | ---- | ---- | | ResNet-34 | 25.7 | 8.3 | 222.5 | 86.0 | | WideResNet-50 | 268 | 68.1 | 44.6 | 86.6 | Best regards! Authors of MambaAD.
Question on the Uni-Medical dataset
Thank you for the clarification. I noticed that the authors referenced [46] for the Uni-Medical dataset. However, the paper in [46] neither includes experiments on medical anomaly detection nor references any medical anomaly dataset. Could the authors please confirm the accuracy of this citation for the Uni-Medical dataset?
Uni-Medical dataset reference.
Thank you for your response. The file **./data/README.md in the official code directory** provided in the abstract of paper [46] documents that **Uni-Medical originates from [A]**. After extracting the Uni-Medical dataset and comparing it with the BMAD dataset proposed in [A], we found that the authors of **[46] selected 3 out of the 6 medical anomaly detection datasets mentioned in [A]. These selected datasets contain both pixel-level and image-level ground truth and were formatted to conform to the MVTec-AD dataset standard**. We have also identified this issue and *attempted to contact the authors of [46] via email to point out the discrepancies between the code and the paper*. **In the revised version, we will cite both [46] and [A] for the Uni-Medical dataset**. Once again, thank you for your efforts in improving the quality of the manuscript. [A] Bmad: Benchmarks for medical anomaly detection. Bao *et al.*. CVPRW’24.
Thanks for the rebuttal
I appreciate the authors' rebuttal and their engagement with my comments. Given that the experiments were conducted on multiple datasets, could the authors clarify which specific dataset was used to obtain the ablation mAD in the table above?
Response to Reviewer FXx2 about mAD.
Dear reviewer FXx2: Thank you for your comments. The ablation experiments in the three tables mentioned in the rebuttal were all conducted on the **MVTec-AD** dataset. **In the PDF, only Tab. 4 and 6** include experiments conducted on **multiple datasets**, as indicated in the first row of the respective tables. **All other tables' ablation experiments** were performed on the standard **MVTec-AD dataset**. Best regards! Authors of MambaAD.
Supplementary Ablation Experiments on VisA dataset.
**Q: Dataset Ablations** To further validate the reliability of the ablation experiments, we have added ablation experiments on the **VisA dataset on mAD** evaluation metric. Due to time constraints, we only conducted ablation experiments on **Tab. 1, 2, 3, 9, and 10 from the PDF on the VisA dataset**. The results on the **VisA dataset are consistent with the conclusions drawn from the MVTec-AD dataset**. We will provide more detailed and comprehensive experiments as supplementary material in the final version. ***Tab. 1 Decoder Ablations*** | Method | Params | FLOPs | MVTec-AD | VisA | | ------------------------- | --------- | -------- | -------- | ---- | | CNN-based | 13.0M | 5.0G | 81.7 | 72.5 | | Transformer-based | 27.1M | 8.4G | 80.2 | 67.4 | | Mamba-based | 22.5M | 7.5G | 82.1 | 72.9 | ***Tab. 2 Scaning Directions Ablations*** | Scan Directions | Params | FLOPs | Training Time | Training Memory | MVTec-AD | VisA | | --------------- | --------- | -------- | ------------- | --------------- | -------- | ---- | | 2 | 21.8M | 7.2G | 2h1m | 6230 | 85.8 | 78.5 | | 4 | 23.1M | 7.5G | 2h56m | 7974 | 85.9 | 78.6 | | 8 | 25.7M | 8.3G | 5h39m | 11802 | 86.0 | 78.9 | ***Tab. 3 Components Ablations*** | Basic Mamba | LSS | HSS | MVTec-AD | VisA | | ------------------- | --- | --- | -------- | ---- | | √ | | | 82.1 | 72.9 | | √ | √ | | 84.9 | 78.0 | | √ | √ | √ | 86.0 | 78.9 | ***Tab. 9 Branches Ablations*** | Method | Params | FLOPs | MVTec-AD | VisA | | ------------ | --------- | -------- | -------- | ---- | | Local | 13.0M | 5.0G | 81.7 | 72.5 | | Global | 22.5M | 7.5G | 82.1 | 72.9 | | Global+Local | 25.7M | 8.3G | 86.0 | 78.9 | ***Tab. 10 Mamba-based Backbone Ablations*** | Backbone | Params | FLOPs | MVTec-AD | VisA | | ----------------- | ------ | ----- | -------- | ---- | | EfficientVMamba-T | 6.3M | 1.0G | 72.7 | 71.2 | | VMamba-T | 30.2M | 6.0G | 82.9 | 76.2 | | ResNet34 | 8.2M | 4.0G | 86.0 | 78.9 |
Please let us know whether we address all the issues
Dear reviewer, Thank you for the comments on our paper. We have submitted the response to your comments. Please let us know if you have additional questions so that we can address them during the discussion period. We hope that you can consider raising the score after we address all the issues. Thank you! Best Regards! Authors of # Mamba AD
Any more comments or concerns?
Dear reviewers, Thanks a lot for your effort in reviewing this submission! We have tried our best to address all the mentioned concerns/problems in the rebuttal. Feel free to let us know if there is anything unclear or so. We are happy to clarify them. Best regards! Authors of MambaAD.
Decision
Accept (poster)