Response to Reviewer eNcJ
Thank you for your valuable feedback to help us improve our paper. We have revised our paper based on your feedback. We detail our response below and please kindly let us know if our response addresses your concerns.
>**Q1**: While the proposed automated evaluation metric using a fine-tuned LLM is novel and promising, details on the construction of the dataset used for fine-tuning are missing in the paper.
**A1**: We randomly select examples from each field for human annotation, including the original inputs (images and questions), ground truth, and responses from evaluation models. For each category, we develop comprehensive and detailed criteria with scoring standards. We first annotate 20 examples, providing specific examples for each score as references. To facilitate the annotation process, we design a graphical annotation tool. Finally, we create a dataset of 800 examples with evaluation scores through human annotation, for fine-tuning the scoring model. We have revised the paper and put these details in Appendix A.3.
****
>**Q2**: The claim L110 “The proposed scoring model ... has proven to be comparable to human evaluation.” is not well justified in its current form. For instance, while Table 5 shows that their proposed method has better similarity with human scoring, further details on how the human annotations were obtained are missing.
**A2**: We use the same construction method as the fine-tuning dataset to randomly select 200 samples with human-annotated scores for evaluating MMIE-Score. We then implement traditional image-text consistency metrics (e.g., CLIPScore) and LVLM-generated scores, then evaluate their scoring quality using several correlation metrics (e.g., Cosine Similarity, MSE, MAE, Pearson) based on human-annotated scores. As shown in Table 5, MMIE-Score outperforms all the above metrics, maintaining the best performance across the board. We have revised the paper and put these details in Appendix A.3.
****
>**Q3**: The effectiveness of the proposed evaluation metric is a function of the evaluation model. Please provide further details on the fine-tuning method and dataset utilized for the evaluation model.
**A3**: We randomly select 800 processed examples from each field for human annotation. As is shown above, this dataset includes the original input (images and questions), ground truth, and responses from evaluation models. To facilitate the annotation process, we design a graphical annotation tool, as shown in the figure below. In this tool, the samples are displayed on the left, while a floating panel on the right allows annotators to score the samples. Annotators can check different criteria, and the final cumulative score is calculated accordingly. Checking a criterion indicates that the sample meets that specific criterion. We have revised the paper and put these details in Appendix A.3.
****
>**Q4**: To better understand the role that the evaluation model plays in the pipeline, please provide qualitative examples of eval model responses corresponding to Fig 7,9,10,11,12 in the appendix.
**A4**: The evaluation model provides detailed and comprehensive feedback for each example based on the criteria and ground truth. We present the corresponding feedback responses for all the cases. We have revised the paper and put these details in Appendix C (see Figure 8,9,10,11,12,13,14,15).
****
>**Q5**: Please provide details on how the human annotations were obtained for Table 5 to better support the claim in L110.
**A5**: We design a graphical annotation tool to facilitate the process. Human annotators score the samples by checking options on the floating panel on the right and assigning a final score. Each option represents a specific criterion, and checking the box indicates that the model's response meets that criterion. Specifically, the "emotional support" criterion is checked by default; if unchecked, the score is reduced by 1 point, as explained in Appendix A.3 of the paper. The final score is determined by the number of checked criteria. We strictly follow this scoring process to build our dataset. Using 800 samples from this dataset, we fine-tune the Intern-VL2 model to enhance its scoring capabilities in our task. We then evaluate its performance by comparing the model's scores with human evaluation scores on 200 samples, demonstrating the effectiveness of our scoring model. We have revised the paper and put these details in Appendix A.3.