Summary
The paper introduce LLaVA-Critic, an open source LMM that has been trained as an evaluator rather than a generator of content. To do so, the paper uses a variety of datasets that include prompt (image+question), responses, and their evaluation. Throughout most of these datasets, the evaluations are done be GPT-4o, and LLaVA-Critic is trained to imitate these evaluations through a standard cross-entropy loss. The model is trained both for point wise evaluation (i.e. LLM-as-a-Judge), where the model needs to output a score between 1 and 10, and pairwise comparisons, where the model has to output a preference between two candidate responses. The experimental results focus on these two settings as well, where results show that the proposed model closely matches the performance of GPT-4o for both settings, and that it performs better than open-source alternatives within a DPO algorithm pipeline.
Strengths
The presentation of the paper is good, with authors being clear and detailed about the procedure being followed. For example, the references to previous datasets, prompts are either well cited in the main paper or detailed in the appendix. Figure 1 also helps understanding what constitutes the majority of the training dataset. The paper mentions that their method is open-source, which indicates that the weights will be released - a net positive for the community. Most of the claims in the empirical section are appropriate with respect to the obtained results, which indicates a fair evaluation from the authors.
Weaknesses
The strongest concern about the paper is that there is nothing that is particularly surprising, or insightful. In essence, the procedure describes entails to distilling GPT-4o into an open-source model using standard cross-entropy, and showing performance that is close to that of GPT-4o. Although this is great for the community, I am struggling to see what the contribution is from a scientific perspective. I can see how this paper would be valuable within a Datasets & Benchmarks track as the one available in NeurIPS, which would likely be a better fit. For a scientific paper, I think there is a necessity to try to answer some question, improve some empirical understanding, or present a surprising phenomenon. At the moment, everything is very as-expected and very standard, ressembling a technical report.
For the preference learning experiments, why isn't there a comparison with GPT-4o directly expressing preferences to create a reward model for DPO? This seems like the more appropriate comparison rather than using GPT-4V for preference alignment. Also, the authors claim that their approach is superior to other competing methods, but isn't this expected given that the other approaches do not fine-tuning from a teacher (i.e. GPT-4o preferences)? Aren't most of the other baselines simply complimentary to fine-tuning on a teacher?
Questions
For the MLLM-as-a-Judge benchmark, how much overlap is there between that benchmark and the ones used to train the model? These experiments are referred to as "out-of-domain", but this is not clearly shown in the paper.
In Table 1, "dominant" might be a typo.
The references for generating reward signals in RL could be better chosen, especially as o1 is a closed source model.