Unraveling and Mitigating Safety Alignment Degradation of Vision-Language Models

Response to Reviewer yyM3

We appreciate the reviewer's insightful feedback. We provide a detailed response below to address the concerns and questions raised by the reviewer. >**W1: Novelty** We fully acknowledge that feature interpolation itself is not novel, as it has been applied in various domains to address distinct challenges, including VLM hallucination[1]. However, the novelty and core contribution of our work lies in identifying and articulating the specific insight that multi-modal representation shifts are a key factor behind safety alignment degradation in VLMs. Feature interpolation emerges as a natural and effective method to act on this insight, enabling us to directly address the representation shift. Our experiments demonstrate that this approach significantly mitigates safety degradation while preserving model utility, validating its efficacy in this context. We believe this targeted application of feature interpolation, guided by our unique insight, contributes meaningfully to the understanding and enhancement of VLM alignment. [1] Favero, Alessandro, et al. "Multi-modal hallucination control by visual information grounding." *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*. 2024. --- >**W2: Safety Judgement** Thank you for your valuable suggestion. Due to licensing and accessibility constraints, we were unable to use LLaMA-Guard for evaluation in this study. However, we acknowledge the importance of employing more reliable models for evaluation and plan to integrate GPT-4 into our future assessments. Regarding human verification, we have not conducted manual evaluation in this study due to resource limitations. That said, we agree that human verification would provide additional assurance of the accuracy and safety of the outputs and is a meaningful addition for future iterations of this work. --- >**Q1: Broader range of alpha values** Thank you for the suggestion to analyze a broader range of alpha values, such as alpha ∈ {1, 10, 100}. However, as demonstrated in our experiments, when alpha reaches a value as small as 2.0, the model begins to malfunction due to overcorrection. This overcorrection causes the representations to deviate excessively from the intended distribution, undermining both safety alignment and the model's general utility. We have provided a visualization in Figure 3 that illustrates this phenomenon, showing how higher alpha values push the representations too far, leading to degraded performance. Expanding the range to include alpha values significantly larger than 2 would, therefore, not yield meaningful results, as the model's alignment capabilities are already compromised at these lower thresholds. Instead, we chose a finer interval within a practical range (e.g., 0.1) to provide a more precise analysis of the optimal alpha value for effective calibration without overcorrection. --- >**Q2: Performance of a pure LLM** Thank you for the valuable suggestion. We agree that presenting the performance of a pure LLM, such as Vicuna, on the VLSafe and JailbreakLLMs datasets would provide valuable context and further illustrate the issue of safety degradation. Including these results would allow for a clearer comparison between the safety alignment capabilities of the original LLM backbone and the full VLM, helping to highlight the extent of degradation introduced by the integration of the vision modality. We will incorporate these results in future revisions to enhance the clarity and comprehensiveness of our analysis. Thank you again for this constructive feedback.

Response to Reviewer UVRU

We appreciate the reviewer's insightful feedback. We provide a detailed response below to address the concerns and questions raised by the reviewer. >**W1: Ideal distribution of MLLMs** Thank you for your thoughtful comment and for highlighting an important perspective on the alignment of MLLMs. We acknowledge that the introduction of multi-modal data, additional parameters, and distinct learning techniques naturally shifts the representation distribution, aligning it more towards multi-modal functionalities rather than the original textual alignment of the LLM backbone. Our study does not argue that the ideal distribution of MLLMs should strictly adhere to the safely trained LLM backbone; rather, we propose that the inherent safety alignment mechanisms of the LLM backbone, which have been optimized for textual inputs, provide a strong foundation for mitigating safety risks in VLMs. This is particularly crucial as the incorporation of vision data has been shown to degrade safety alignment. Our methodology seeks to recalibrate multi-modal representations to a space where the LLM's intrinsic safety mechanisms can be effectively leveraged, without compromising the model’s multi-modal functionalities. While the safety regulations of the LLM may not apply directly to the shifted inputs, our empirical results demonstrate that partial recovery of safety alignment is achievable. We appreciate your observation about the need for broader evidence, and we agree that future work should investigate the balance between maintaining safety and fostering comprehensive VLM development, possibly through hybrid alignment strategies that integrate both safety and utility optimization. --- >**W2: Utility Benchmarks** Thank you for this constructive comment. We provide the results on MMMU as follows. | | MMMU-Overall | |----------------|--------------| | LLaVA-v1.5-7B | 33.90 | | + CMRM | 33.00 | | LLaVA-v1.5-13B | 36.30 | | +CMRM | 35.80 | These results are consistent with the observations reported in Table 1, where CMRM demonstrates minimal impact on utility performance while significantly improving safety alignment. Importantly, the marginal differences in utility scores highlight the effectiveness of CMRM in preserving the functional capabilities of the models across different benchmarks. We recognize the importance of extending utility evaluation to other comprehensive benchmarks, such as MM-Vet and MME, to further validate the robustness of our approach. As such, we are actively working on completing experiments across all settings outlined in Table 1 and will include these results in the revised version of the paper. --- >**W3: Utility Benchmarks** Thank you for this suggestion. We will clarify the details when introducing the baseline methods. According to the work of VLGuard [1], both VLGuard Mixed and VLGuard PH train VLMs on a set of safety data and update full parameters. We evaluate the safety performance of these baseline methods by directly testing the models released by [1]. [1] Safety Fine-Tuning at (Almost) No Cost: A Baseline for Vision Large Language Models --- >**Q1: Recover safety alignment knowledge** Thank you for your thoughtful comment regarding the relationship between safety alignment degradation and the potential forgetting of safety alignment knowledge in MLLMs (VLMs). Our study specifically investigates the safety alignment degradation in VLMs by comparing the safety alignment capability of the **entire VLM** to that of its **updated LLM backbone** within the VLM. During the instruction fine-tuning process, both the vision encoder and the LLM backbone parameters are updated, which inevitably impacts the pre-existing safety alignment knowledge of the original LLM backbone. Therefore, the degradation we address is not the comparison between the VLM's safety and the original standalone LLM's safety, but rather between the safety of the VLM as a whole and the updated LLM backbone embedded within it. As such, the goal of our proposed CMRM is to restore the safety alignment of the VLM to the level of its updated LLM backbone, which inherently represents the most aligned state achievable within the VLM after fine-tuning. Regarding the broader question of whether forgotten safety alignment knowledge can be recovered through training-free calibration, we agree this is an intriguing direction for future research, as it would open new pathways for enhancing safety alignment without additional fine-tuning.

Response to Reviewer 8DSj

We appreciate the reviewer's insightful feedback. We provide a detailed response below to address the concerns and questions raised by the reviewer. > **W1: Selection of meaningless or corrupted images** Thank you for pointing this out. In our study, we used the original image as it is, without introducing corruption or modification, for simplicity and to maintain consistency across experiments. This approach allowed us to directly evaluate the model's response to unaltered multi-modal inputs. While exploring corrupted images, such as those with Gaussian noise, could provide further insights, our goal in this pilot study was to establish the feasibility of the proposed method using straightforward input configurations. We appreciate your suggestion and consider it a valuable direction for future work. > **W2: Generalization on OOD images** We appreciate your insightful question regarding the generalizability of the calibration term to OOD images and broader safety tasks. Due to licensing and accessibility constraints, we were unable to directly test our approach on popular benchmarks such as MM-SafetyBench and FigStep, which we acknowledge as valuable resources for future validation. To partially address generalization, however, we point to the results on ScienceQA (Table 1), where the images come from a distinct distribution compared to VLSafe. These results demonstrate that our method retains competitive performance when evaluated on a different dataset, indicating its potential for generalization beyond the original anchor dataset. We agree that evaluating our method on diverse tasks and datasets like stable-diffusion-generated images and MM-SafetyBench subtasks would strengthen the claims of generalization. This remains an important direction for future work, and we appreciate your suggestion to further investigate this aspect. > **W3: Utility Test** Thank you for this constructive comment. We provide the results on MMMU as follows. | | MMMU-Overall | |----------------|:--------------:| | LLaVA-v1.5-7B | 33.90 | | + $CMRM_{dataset}$ | 33.00 | | LLaVA-v1.5-13B | 36.30 | | + $CMRM_{dataset}$ | 35.80 | These results are consistent with the observations reported in Table 1, where CMRM demonstrates minimal impact on utility performance while significantly improving safety alignment. Importantly, the marginal differences in utility scores highlight the effectiveness of CMRM in preserving the functional capabilities of the models across different benchmarks. We recognize the importance of extending utility evaluation to other comprehensive benchmarks, such as MM-Vet and MME, to further validate the robustness of our approach. As such, we are actively working on completing experiments across all settings outlined in Table 1 and will include these results in the revised version of the paper. > **W4: Evaluation Model** Thank you for your valuable suggestion. Due to licensing and accessibility constraints, we were unable to use LLaMA-Guard for evaluation in this study. However, we acknowledge the importance of employing more reliable models for evaluation and plan to integrate GPT-4 into our future assessments. Regarding human verification, we have not conducted manual evaluation in this study due to resource limitations. That said, we agree that human verification would provide additional assurance of the accuracy and safety of the outputs and is a meaningful addition for future iterations of this work. > **W5: Missing Relevant Methods** Thank you for the valuable suggestion. We acknowledge the importance of including a broader range of relevant works in the related work sections to provide a comprehensive view of VLM safety alignment. We will expand the discussion to incorporate additional training-based and inference-based methods for safety improvement.