`>>> Q1-1`: Joint Optimization
We have attempted to optimize both image and text perturbations jointly, but the performance of the joint optimization is worse than that of the proposed VLAttack.Toward the joint optimization, we modify the proposed BSA loss by adding a new loss term for the text encoder, and its format is similar to the image encoder and multimodal encoder, where we compare the block-wise similarity between the clean text input and the perturbed text input.
At each attack step, we directly follow the current BSA design to generate image perturbations for the image modality. We then move to the text modality. Due to the discrete nature of text data, we follow existing work [21] using the word embedding space to generate the text perturbations. First, we use the modified loss to calculate the gradients on the text and then add them to the original word embeddings. The summation can be treated as the ideal word embeddings of the perturbed text. The next step is to map these continuous embeddings to discrete words. For each informative word in the original text, we use BERT to calculate its synonym set following [20]. We then use the word synonym substitution approach to replace the original words with their synonyms under the constraint listed in Eq. (1) in the original paper to guarantee the semantics of the new perturbation.
The new image and text perturbations will be iteratively updated until $N$ steps with the above processes. The experimental results are shown in the following table:
|Dataset| BSA| BSA-joint| VLAttack|
|-|-|-|-|
|VQAv2|65.20|66.48|__78.05__|
BSA-joint denotes the joint optimization, which performs slightly better than the proposed BSA on the VQAv2 dataset but is still far inferior compared to VLAttack. This is because the input text $\mathbf{T}$ is relatively short, and frequently perturbing texts and strictly coupling the updates may lead to fluctuations in the gradient information. Therefore, joint optimization degrades the overall attack performance.
`>>> Q1-2`: BSA v.s. BadEncoder
We admit the overall idea of the proposed BSA and BadEncoder is similar, and they all use similarity as the optimization target. However, the proposed BSA is different from or better than BadEncoder.
(1) BadEncoder (Eq.2--5) only utilizes the final output feature vectors from the whole encoder, ignoring the outputs from the intermediate layers/blocks. However, BSA calculates fine-grained similarity scores. As shown in Eq. (2) of our original paper, we distinguish the outputs from image and Transformer encoders. Such a design can modify the low-level vision features and the high-level cross-modal representations. In our setting, we attack fine-tuned models of a wide diversity of downstream tasks, including but not limited to image classification tasks like Badencoder. The parameters of these task-specific models are fully finetuned on distinct datasets, and the output representations of the encoder significantly change accordingly. Thus, instead of only attacking the output feature from the last layer like BadEncoder, perturbing each intermediate feature representation from each encoder and each block can enhance the attack performance. This statement is also verified in Section 5.4, where the ASR score of BSA is higher than only attacking a single encoder.
(2) The motivation for adopting the cosine distance is different. In BadEncoder, CLIP uses cosine similarity as a loss function to calculate distances for positive/negative image text pairs, which is motivated by the pre-training strategy of CLIP. However, we adopt cosine similarity because the fine-grained token representations attend to each other in the inner product space, which is inspired by the mechanism design of the Transformer structure. This motivation is also illustrated in Lines 180-181 of our original paper.
We will add these discussions in the final version.
`>>> Q2`: CLIP and BLIP
Thanks for the constructive suggestions. We agree that adding more experiments on CLIP and BLIP is beneficial for validating the contributions of our work more comprehensively. For the CLIP model, we will add the experiments through the image classification task, which has been discussed in the previous response on the SVHN dataset. We will add results using more datasets.
For the BLIP model, we experiment with the VQA task of the BLIP model using the VQAv2 dataset. The proposed models still achieve better performance, as shown below:
|Dataset|DR|SSP|FDA|BSA|B&A|R&R|Co-Attack|VLAttack|
|-|-|-|-|-|-|-|-|-|
|VQAv2|7.04|11.84|7.12|26.36|15.30|2.94|20.62|__45.64__|
We will add all experimental results in our revised version to enhance the validation of model effectiveness.
We sincerely appreciate your valuable feedback, which elevates the significance of our model's design, highlights its distinctions from existing approaches, and fortifies our work's overall quality. We hope our responses can adequately address your concerns.