IDA-VLM: Towards Movie Understanding via ID-Aware Large Vision-Language Model

Thank you for your positive review and constructive suggestions. **Weakness**: ID images is identical to the region of interest (ROI) in the original image. This is true for the first stage, as mentioned in line #255, while the second-stage fine-tuning data utilizes ID images from different frames of movies, ensuring that characters appear in varying states between ID images and test images. The model need to memorize essential ID features for recognition rather than merely comparing identical content. **Question1**: construct localization samples. These are some inspiring ways to develop localization samples, making ID images different from the corresponding characters in test images. In our work, we directly use MovieNet to construct localization samples that meet requirements. **Question2**: multiple ID images and multiple test images. When there are multiple ID and test images, concatenate and repeat queries of all ID images, then use them to conduct cross-attention with each test image. That is, the test images should attend to all ID queries. In this cross-attention, the shape of ID queries is (N_testimg, N_idimg x L_query, d), and test images' features are (N_testimg, L_img, d). **Ethics concern**: We will discuss the copyright of these downloaded images in the ethical considerations section. When releasing the dataset, we try to provide image URLs rather than directly sharing the image files.

Thank you for your valuable review and suggestions. **Weakness1**: name samples. The name is randomly sampled from a predefined set, ensuring unique names for different characters within each test image. **Weakness2**: the results of Qwen-VL-Chat In Table2 and Table3, we only compare our model with closed-sourced models, because other open-sourced models (like Qwen-VL-Chat) have poor performance, which stems from their lack of ID-instruction learning. **Weakness3**: More models. Thanks for the suggestion. We didn't evaluate LLaVA, because the models selected for testing should support multiple images input. We will test some new open-sourced VLMs in the next version and add them to the paper later. **Weakness4**: more standard benchmarks. We currently utilize MME, SEED-Bench for evaluation, which demonstrates our model with ID instruction tuning keeps basic multimodal ability. We did not emphasize evaluation on general multimodal benchmarks (OCR, reasoning), as our primary focus was on identity-related capabilities. **Issue1**: We will increase the font size in figures.

Thank you for your careful review and questions. **Weaknesses1**: share a similar template. This template format serves solely to inform the model about ID information, after which queries about test images can be flexible, including various questions and descriptions. The ID template functions similarly to a system prefix prompt in instruction tuning, remaining constant across interactions. Furthermore, the specific format of conveying ID information is not crucial, as we can preprocess any combination of ID role names and corresponding images into our training template. **Weaknesses2**: the increase in performance. They share the similar template. The similar templates are designed to help the model memorize the ID recognition task structure, serving as a unified format to convey ID information to the model. Performance improvements stem from our constructed instructional data that activates the model's ID perception capabilities. **Weaknesses3**: Details of the ID-Former when multiple ID images. Normal cross-attention in Q-Former uses learnable queries as Q, and image features as K, V. In contrast, the additional cross-attention (the pink one in ID-Former architecture image) in ours uses test image features as Q, and the output queries of ID images as K, V, that is, using semantic features of ID images to modulate the input embeddings of test images. When there are multiple ID and test images, concatenate and repeat queries of all ID images, then use them to conduct cross-attention with each test image. In this cross-attention, the shape of ID queries is (N_testimg, N_idimg x L_query, d), and test images' features are (N_testimg, L_img, d). **Question**: the fixed template The fixed template serves only a task format to instruct the model. In the real-world VQA samples, if you have ID role names and corresponding images, you can preprocess them into the template.

Dear reviewer nMGR, We sincerely appreciate your review of our work and your valuable contribution to improving the quality of the paper. Have we effectively addressed your concerns in our rebuttal? If you still have any issues or new concerns, please feel free to let us know so we can continue the discussion.

Thank you for your in-depth review and questions. **Weaknesses1**: Details of the ID-Former. Compared to general Q-Former, our ID-Former adds an additional cross-attention module, which is the pink one in ID-Former architecture image (Figure 2). Normal cross-attention in Q-Former uses learnable queries as Q, and image features as K, V. In contrast, the additional cross-attention in ours uses test image features as Q, and the output queries of ID images as K, V, that is, using semantic features of ID images to modulate the input embeddings of test images. When there are multiple ID and test images, concatenate and repeat queries of all ID images, then use them to conduct cross-attention with each test image. **Weaknesses2**: The integration of LLaVA and ShareGPT4V data. Due to the space limitation of the main paper, we put the ablation about the mixing rate of LLaVA and ShareGPT4V data in Appendix B (Table 10), as indicated in line #523. We will try to adjust the space and move the ablation to the main paper. The results of "IDA-VLM (w/ LLaVA data only)" may be close to results of Qwen-VL-Chat, because LLaVA data is general multimodal instruction tunign data, which can't unleash the ID-awareness ability of LVLM. Moreover, without ID instruction tuning data, IDA-VLM can't optimize ID-Former, so the result model will be similar to the baseline, Qwen-VL-Chat. **Weaknesses3**: The experimental setup primarily addresses scenarios where reference ID count equals or is less than the character count in test images. This is an interesting question, because on the one hand, characters of interest may not be in the test images. On the other hand, additional ID images may bring interference and make ID recognition harder. Actually, when we construct MM-ID benchmark, we also consider this problem to make more diverse test sampels. For example, in some questions of animation, we give more ID images than characters appearing in the test image. We added such examples of MM-ID in the new version of Appendix. In some easy Q&A samples, IDA-VLM can give right answers, but in caption sub-task, IDA-VLM may generate content of non-existing characters. This is bacause in our training data, the number of ID images generally equals to the character count in test images. If we add additional ID images to training samples, this issue can be alleviated. **Weaknesses4**: A systematic evaluation of robustness to ID variations. This is an interesting and meaningful question. In our test samples, the ID images are clear, enabling the model to capture complete identity characteristics. The test images exhibit various ID variations, including pose and clothing changes, demonstrating our model's robustness to identity variations. We included detailed case studies of ID robustness in the appendix, analyzing model performance under different ID variations. We observe that the model demonstrates robust performance against variations in clothing and viewing angles. The primary challenge arises from interference by IDs with similar features in the test images.

**Question1**: The potential scoring variance inherent in subjective assessments with wide scoring ranges by GPT-4. Empirically, GPT-4's scoring aligns well with human subjective assessments. The following examples demonstrate GPT-4's scoring capability, where it provides reasonable explanations for its assessments while maintaining good discriminative ability across adjacent scores. Example1: consistently provide reasonable explanations > "prediction": "Caption Mitch is standing with his arms crossed.", "GT": "Caption Mitch is standing, crossing his arms and looking at others.", > "review1": "8\n\nThe prediction and the GT are largely consistent in conveying the main action that Captain Mitch is performing - standing with his arms crossed. Though the GT provides a bit more detail by including that he is also looking at others, this additional information does not change the overall action significantly. The slight difference is considered a minor detail and does not impact the understanding of Caption Mitch's posture or demeanor in a substantial way. The main action (standing with arms crossed) is captured correctly in both responses. Hence, the prediction is relevant, fairly accurate, and could be helpful, missing only the extra detail of where Captain Mitch is looking." > "review2": "9\n\nThe prediction and the GT are consistent in describing Caption Mitch's posture and action. Both state that Caption Mitch is standing with his arms crossed. The GT adds that he is looking at others, which is a minor detail not mentioned in the prediction but does not fundamentally change the understanding of Caption Mitch's stance or action. Thus, the additional detail in the GT is not substantial enough to significantly affect the quality of the answer. Therefore, the prediction is largely accurate and relevant, earning a high score. However, due to the slight omission of what Caption Mitch is looking at, a perfect score is not warranted." Example2: discriminative ability across adjacent scores. > "GT": "Barbara, Violet, Mattie are in a living room and discussing intensely. Barbara is standing and the other two sitting. They all hold some cards in their hands and look at them.", > "prediction": "Barbara stands in a room holding some papers, facing away from the camera. Mattie sits to the right, looking at papers in her lap. Violet sits to the left, also looking at papers.", > "score": "6", > "prediction": "Barbara stands in a room holding some papers, facing away from the camera. Mattie sits to the right, holding papers and looking down. Violet sits to the left, also holding papers and looking down. They appear to be in a discussion.", > "score": "7", **Question2**: ID matching performance. The decline in matching performance with first-stage fine-tuning can be attributed to the absence of matching tasks in the first-stage dataset, resulting in a dilution of matching-related training data. Better matching performance could be achieved by adjusting the proportion of task-specific data.