Sm: enhanced localization in Multiple Instance Learning for medical imaging classification

Questions regarding the new experiments

I would like to thank the authors for extending their experiments and including new insights. I would appreciate the authors if they could provide more details regarding the following questions. It has been mentioned in line 45 that "This is a flexible module that can be used alone on top of classical MIL approaches, or in combination with transformers to also account for global dependencies." If I have understood it right, the main contribution is to have sm as a local-global module that helps to improve performance by capturing both local and global. Judging only from Table 1 and Resnet50 with ImageNet weights, how do authors justify the models without global interactions that are superior to their proposed smTAP with global interactions? If I am not missing anything, this is a bit contradictory to the claims of the paper. Also, comparing Resnet50+BT and Resnet50 only for the Calmelyon16 dataset, it seems that their proposed model benefits heavily from pre-trained features (improvement comes mostly from the encoder rather than the aggregation proposed). I am wondering if the authors can justify this observation. Does this come from any biological underlying meaning or can the Sm method be intuitively linked to anything?

Please read the rebuttal to check if the authors addressed your concerns

Dear Reviewer uFQt, Can you have a look at the rebuttal and see if your concerns have been addressed? Best regards Your AC.

On the new questions

We would like to thank Reviewer uFQt for their positive comments on the new experiments and the new insights we have provided. We address the new questions below. > **Q. It has been mentioned in line 45 that "This is a flexible module that can be used alone on top of classical MIL approaches, or in combination with transformers to also account for global dependencies." If I have understood it right, the main contribution is to have sm as a local-global module that helps to improve performance by capturing both local and global. Judging only from Table 1 and Resnet50 with ImageNet weights, how do authors justify the models without global interactions that are superior to their proposed smTAP with global interactions? If I am not missing anything, this is a bit contradictory to the claims of the paper.** As we indicate in the paper, the proposed Sm operator introduces local interactions in a principled manner (through Dirichlet energy minimization with the presented theoretical guarantees). This operator can be naturally combined with self-attention layers (transformer layers) to account for both local and global interactions. As the reviewer points out, in Table 1 of the rebuttal PDF, when looking at ResNet50 with Imagenet weights, the models without global interactions (i.e., no transformer layers) do better at the instance level (localization task), with the proposed SmAP achieving the highest rank (see the table in the rebuttal to Reviewer uFQt). However, when looking at the classification task (bag level, Table 2), the models with global interactions outperform to those without global interactions, with the proposed SmTAP achieving the highest rank (see the table in the rebuttal to Reviewer uFQt). These results suggest that models without global interactions (SmAP) perform better at the instance level (localization task, Table 1), while models with global interactions perform better at the bag level (classification task, Table 2). This behaviour was also pointed out by Rev. YAE1. As we argue in the rebuttal to Reviewer YAE1, this is explained by the fact that self-attention layers of transformers obtain a richer bag representation (which leads to better bag level results). However, the signal that identifies each token in the sequence becomes weaker after each self-attention layer (the tokens become more similar). This leads to worse instance level results. Note that this phenomenon is known as over-smoothing and has been observed in (1), (2), (3) (see references below). We believe that the results in Table 1 are not contradictory with the claims in the paper, but rather reinforce them. The proposed Sm improves instance level results (localization) and remains competitive at the bag level (classification). This holds true whether it is combined with transformer layers (SmTAP) or not (SmAP). > **Q. Also, comparing Resnet50+BT and Resnet50 only for the Calmelyon16 dataset, it seems that their proposed model benefits heavily from pre-trained features (improvement comes mostly from the encoder rather than the aggregation proposed). I am wondering if the authors can justify this observation. Does this come from any biological underlying meaning or does the Sm method can be intuitively linked to anything?** As the reviewer points out, the proposed Sm benefits from SSL pre-trained features. Our intuition is that if an SSL pre-trained encoder is used, instances that are similar from a biological point of view will tend to cluster together in the feature space. In particular, instances with the same label will be clustered together, and far away from instances with a different label. Since neighboring instances are naturally (a priori) expected to have similar biological properties (e.g., the same label), the smoothness introduced by the proposed Sm will be particularly beneficial. This interesting observation is related to that in CAMIL [13], where it was shown that a SSL pre-trained encoder is crucial for the optimal performance of their neighbor-constrained attention mask. ---- (1) Shi, Han, et al. "Revisiting Over-smoothing in BERT from the Perspective of Graph." International Conference on Learning Representations (ICLR). 2022. (2) Wang, Peihao, et al. "Anti-Oversmoothing in Deep Vision Transformers via the Fourier Domain Analysis: From Theory to Practice." International Conference on Learning Representations (ICLR). 2022. (3) Dong, Yihe, Jean-Baptiste Cordonnier, and Andreas Loukas. "Attention is not all you need: Pure attention loses rank doubly exponentially with depth." International Conference on Machine Learning (ICML). PMLR, 2021.

Concern whether our reply was satisfactory

In case it is necessary, we want to summarize our response to the new questions by the reviewer. > **Q. It has been mentioned in line 45 that "This is a flexible module that can be used alone on top of classical MIL approaches, or in combination with transformers to also account for global dependencies." If I have understood it right, the main contribution is to have sm as a local-global module that helps to improve performance by capturing both local and global. Judging only from Table 1 and Resnet50 with ImageNet weights, how do authors justify the models without global interactions that are superior to their proposed smTAP with global interactions? If I am not missing anything, this is a bit contradictory to the claims of the paper.** Instance level results worsen with transformers. This is not a consequence of the proposed Sm, but of the oversmoothing phenomenon observed in transformers. This has been studied in the works referenced in our previous response. The proposed method without a transformer outperforms methods that do not a transformer; and the proposed method with a transformer outperforms methods that use a transformer. > **Q. Also, comparing Resnet50+BT and Resnet50 only for the Calmelyon16 dataset, it seems that their proposed model benefits heavily from pre-trained features (improvement comes mostly from the encoder rather than the aggregation proposed). I am wondering if the authors can justify this observation. Does this come from any biological underlying meaning or does the Sm method can be intuitively linked to anything?** Note that every method has access to the same features, and every method benefits from using them (see bag level results in Table 2). That our method benefits at the instance level from using, we interpret it as a sign that local dependencies indeed play an important role in MIL localization. Please let us know if you have any questions so that we can respond in a timely manner. We look forward to your response, Authors

Please read the rebuttal to check if the authors addressed your concerns

Dear Reviewer jUey, Can you have a look at the rebuttal and see if your concerns have been addressed? Best regards Your AC.

reviewer feedback

Thanks for the authors' effort to present additional results and make clarifications. I raised the score despite the concern that this still seems to be a marginal improvement.