Summary
The authors introduce a method for anomaly segmentation that relies solely on visual information and does not require any anomaly training data or language guidance. Their method is based on change segmentation, which allows for the synthesis of large-scale image pairs for training the anomaly segmentation model. The proposed MetaUAS framework demonstrates its generalizability across three industrial anomaly segmentation datasets: MVTec, VisA, and Goods, using one normal image per class as a prompt. Within their framework, the authors extensively experimented with different modules, including an image encoder, feature alignment, and a segmentation decoder. Notably, their feature alignment approach, which is based on a weighted combination of image prompt features, proves to be highly effective for anomaly segmentation by bringing together features of semantic segmentation and change segmentation.
Strengths
- A novel approach is presented by formulating anomaly segmentation as change segmentation.
- Extensive experiments on multiple datasets and in different model configurations demonstrate the effectiveness of the presented method, MetaUAS. In particular, the anomaly segmentation performance benefits from the specifically designed soft alignment module.
- It is shown how efficiently a change segmentation dataset can be created with synthetic data, which is then used for training anomaly segmentation.
- MetaUAS, trained on synthetic data, generalizes well to real-world problems. Notably, the method does not require any anomalous data for training but still outperforms existing methods that use such auxiliary data.
- Unlike other existing methods, MetaUAS does not require guidance from language and solely relies on visual information.
Weaknesses
- At least one normal image per class is always needed as a prompt to perform anomaly segmentation. All prompts must be processed by the encoder to extract their features, which could become expensive as the number of classes increases.
- The prompt features can be stored offline as a prompt pool to reduce forward passes at test time when using MetaUAS. However, it is unclear whether a single example can represent all normal examples for a class effectively (Figure 4 highlights the issue of different prompts). Searching for the best matching normal image for each query image is obviously resource-intensive.
- The anomalies shown in the industrial datasets relate to small changes in the appearance of the objects. The objects are also always displayed in the same scene. It is noted that the orientation of the object affects the anomaly segmentation performance. To further investigate the limitations, it would be interesting to see what would happen if the scenes changed more dramatically, which is realistic in open-world settings. For example, would MetaUAS be robust if the same normal object were shown with a different background?
- Minor: The paper could benefit from better writing and structure. There are a few typos throughout the text, and the figures and tables could be placed differently. It is not always easy to follow.
Questions
- Some more details on the training process would be appreciated. For instance, there is no information on how much training data was required to achieve the presented performance or how robust the model training was. The training data seems to be created in an automated fashion; to what extent does the amount of training data affect the anomaly segmentation performance?
- One could use more than one image as a prompt to potentially obtain more robust anomaly scores. How efficient is the inference? In particular, compared to other existing approaches, could one afford a larger prompt pool or multiple forward passes of MetaUAS?
- Relying on the changes in the visual appearance of objects seems to be a valid approach for anomaly segmentation in industrial settings. The same objects often appear in the same scene, and the anomalies are clear, small visual changes. Therefore, guidance with language seems unnecessary, as the semantic information is not crucial for this kind of anomaly segmentation. However, for other datasets on anomaly segmentation, semantic information seems crucial, such as the [SMIYC](https://segmentmeifyoucan.com/) benchmark, where any obstacle on the road is considered an anomaly. Could MetaUAS also be applied to such more challenging tasks where the normal "object" itself (e.g., the road) could have varying visual appearances as well?
Limitations
The authors have discussed limitations. Potential negative social impact have not been mentioned.