Adversarial patch attacks on vision transformers using transferable gray-box with self-attention

Vision Transformers (ViTs) and their variants have been increasingly adopted in intelligent systems. However, the security implications of partially exposing self-attention information in ViTs remain largely underexplored. Existing adversarial attacks on ViTs typically rely on white-box assumptions with unrealistic access or black-box settings with limited effectiveness, limiting their relevance to restricted deployment conditions. In this paper, we investigate a plausible gray-box threat model motivated by interpretability-oriented deployment scenarios, where limited attention-related information may be observable through interpretability or analysis interfaces. Our analysis suggests that such partial attention exposure may introduce a previously underexplored vulnerability in Vision Transformers. Based on this observation, we propose Targeted Patch Perturbation (TPP), an attention-guided patch perturbation framework tailored for restricted gray-box settings that localizes semantically important patches and restricts perturbations to these regions. The proposed method enables effective adversarial attacks without requiring gradient access to the target model during attack execution. Experiments on CIFAR-10, CIFAR-100, and ImageNet demonstrate that TPP consistently outperforms strong black-box baselines under the same perturbation budget. Further evaluations across multiple ViT architectures reveal that attention-guided patch selection can substantially improve attack transferability under restricted-information settings, suggesting that semantically aligned perturbation localization may affect shared semantic representations across transformer models and influence the adversarial robustness of Vision Transformer systems. The source code is available at: https://github.com/BaoChau-Ho/TGP_adv