We appreciate your insightful viewpoints and thoroughly enjoy the discussion.
> Q1: Looking at the images, I was also considering if a signal-to-noise analysis could provide some insights if the features considered are more related to the foreground than the background.
This is indeed an interesting perspective. We have given consideration to this issue and have come up with a few directions to explore.
If we consider the attribution map on the feature map as a grayscale image, with attribution values as pixel values, features deemed to contribute to predictions (useful features) could be considered as signals.
Under the assumption of the highly effective visual explanation method, these useful features, such as the foreground, should have high pixel values, making them meaningful signals. On the other hand, the background should possess low pixel values, representing insignificant noise. Therefore, we can use the signal-to-noise ratio (SNR) to quantify the uncertainty of the model's attribution explanations:
$$\text{SNR} (
\text{db})= \frac{\text{Signal Power}}{\text{Noise Power}}$$
If the model makes a confident and correct decision, the SNR should be very high. However, this approach also presents some challenges: 1) How do we define the scope of the signal (foreground). We may need a threshold or algorithm to differentiate which regions on the attribution map are background and which are foreground. 2) When there are no in-distribution objects in the entire image (OOD), we should consider the entire image as noise. In our observations, this scenario can result in widely scattered and exceptionally high noise values. We need to design a more appropriate matrix to reflect SNR.
**We consider that this approach could be highly beneficial for object-level OOD detection.** When an image contains multiple objects that need to be recognized, we can consider the regions within the bounding boxes as the signal (foreground) and those outside the boxes as the background. Then we can calculate the SNR on the attribution map to reflect the model's uncertainty about predictions within these regions. This method might be effective in detection scenarios with relatively stable environments, such as autonomous driving, industrial identification, etc.
> Q2: Considering your last comment, I'm not sure I understand the problem of the proposed approach to deal with ViT. Given the growing popularity and effectiveness of transformer architectures, how to adapt or evolve the proposed to accommodate these new advancements?
Given the current trends, providing visual explanations for large models has been a focal point for both the interpretability academic community and the engineering efforts in recent years. Honestly, this is also the research direction we are currently pursuing.
Overall, our approach involves utilizing model-explained prediction uncertainty for OOD detection. For large models, our ongoing research primarily focuses on two directions:
- Direction 1: **For transformer-based model.**
Despite the differences between transformer-based architectures and the convolutional feature layers of CNNs, methods based on attribution gradients can still provide explanations for predictions made by models using transformer structures (such as ViT). For the transformer-based model, we consider the attribution gradients on the attention matrix.
Many traditional CAM methods, including GradCAM, are initially proposed based on CNN structures, and therefore, their performance on transformer-based models might be suboptimal. **However, there is a line of improvements available now to enhance the gradient-based visual explanations on such models**. One prominent example is [1]. Building upon these enhancements, we are currently researching how to identify attribution gradient abnormality on the attention matrix to reflect the model's uncertainty.
Furthermore, the Attention mechanism in transformer-based models can also offer directions for visual explanations. Researching uncertainty in this context can further enhance OOD detection.
[1] Chefer, Hila, Shir Gur, and Lior Wolf. Transformer interpretability beyond attention visualization. CVPR. 2021.
- Direction 2: **For CNN-based backbone.**
Although the mainstream of large models isn't purely CNN-based, many multimodal large models (like CLIP) and downstream tasks (such as object detection) still utilize CNNs as the backbone networks for visual feature extraction.
How to reflect the model's uncertainty in visual feature extraction on these backbones is also a topic we are currently researching.