Robustness of SAM: Segment Anything Under Corruptions and Beyond

Segment anything model (SAM) is notable for its ability to cut out any object and its impressive zero-shot transfer performance with prompt guidance. However, its robustness under corruption remains underexplored. Prior work indicates that SAM is biased towards texture rather than shape. We first evaluate its robustness against style transfer (synthetic corruption), followed by 15 common corruptions, and find that SAM demonstrates robustness in these scenarios. Moreover, we assess SAM's resilience to local patch occlusion and adversarial patch attacks, finding it resilient to occlusion but vulnerable to patch attacks. We observe that patch attacks disrupt the feature map and misguide the attention mechanism, whereas patch occlusion does not cause such misguidance. Given that visible patch attacks are easily detectable, we further investigate robustness against global adversarial attacks that remain imperceptible to humans by employing basic attacks and a new mask-aware loss. Our findings suggest that SAM's vulnerability primarily stems from its more reliance on high-frequency signals compared to low-frequency ones. Moreover, we further investigate the targeted transferability of adversarial examples and propose a contrastive learning-driven approach with a plug-and-play modulation module, enabling more effective transferability across different SAM models. In addition, we investigate a range of defense strategies to enhance the robustness of SAM, as well as stronger and more adaptive attack methods. Overall, this work provides a comprehensive study of SAM's robustness across diverse scenarios and is expected to offer valuable insights into its practical applicability and effectiveness in real-world settings.