Seeing Eye to AI: Human Alignment via Gaze-Based Response Rewards for Large Language Models

Response to Official Review of Submission11182 by Reviewer WBeE

* **W1** First, it is difficult for me to assess whether the improvements on reward modeling reported in the paper are significant. According to Table 2, the test sets used for reward model evaluation are quite small (416 and 364 examples) and I did not find information on the statistical significance of the differences in results. I am also skeptical whether the improvements in scores would be robust to different random seeds and other parameters. It would be beneficial to add statistical significance tests and further discussion on how you ensured your findings are robust. Thank you for this comment. Our experiments involve the rigorous testing of six baselines against six models each (with the exception of Mistral which has three models). In total, we trained and evaluated 6 baselines and 30 models. For 28 out of those 30 models, we have demonstrated significant performance gains compared to their respective baseline. Regarding the robustness of results across different parameters or seeds, and taking into account the computational requirements, we wish to highlight that our methodology implements standard practices in the field, as demonstrated in prior works [Cui et al., 2024; Wang et al., 2024c; Touvron et al., 2023; Köpf et al., 2023]. These studies also report results based on a single seed for models of comparable scale. Currently, our computational resources do not allow us to retrain all models. However, we are in the process of training one configuration with a different seed and anticipate sharing those results before the rebuttal period concludes. Conducting a comprehensive evaluation across multiple seeds for all models would require approximately one month, and we would be prepared to include this analysis in the camera-ready version of the paper, should it be accepted. We have calculated the mean and standard deviation of the differences in validation accuracy between each model variant and the baseline, using the 5 checkpoints closest (in time) to the highest accuracy. We conducted two-sample t-tests calculating test statistics (\(t\)) and p-values (\(p\)) for each comparison. The next two tables show all results to be statistically significant. To control for Type I errors (i.e. false positive cases) due to the multiple comparisons, we applied the two-stage Benjamini-Hochberg (fdr\_tsbh) method to adjust p-values and control the false discovery rate (FDR). **Table: Mean and standard deviation of the difference in accuracy for each combination and the baseline.** The mean is based on the 5 checkpoints in validation for the oasst1 dataset. * indicates significance (**p-values < 0.01 and *p-values < 0.05). | | | **LLaMA-Instruct** | **LLaMA** | **Mistral** | |----------|-------------------|----------------------|---------------------|---------------------| | **Concat** | $fcomb_{1}$ | 8.5 ± 0.8** | 4.5 ± 1.0* | 2.3 ± 0.2* | | | $fcomb_{2.5}$ | 4.7 ± 1.1* | 7.6 ± 0.7* | -0.6 ± 1.5* | | | $fcomb_{2.2}$ | 5.7 ± 1.9* | 4.7 ± 3.1* | -0.3 ± 1.8* | | **Add** | $fcomb_{1}$ | 1.0 ± 0.3* | 2.6 ± 0.3* | - | | | $fcomb_{2.5}$ | 2.4 ± 0.5* | 6.3 ± 0.2* | - | | | $fcomb_{2.2}$ | 12.6 ± 0.2** | 6.0 ± 0.3* | - | **Table: Mean and standard deviation of the difference in accuracy for each combination and the baseline.** The mean is based on the 5 checkpoints in validation for the helpsteer2 dataset. * indicates significance (**p-values < 0.01 and *p-values < 0.05). | | | **LLaMA-Instruct** | **LLaMA** | **Mistral** | |----------------|------------------|----------------------|---------------------|---------------------| | **Concat** | $fcomb_{1}$ | 8.5 ± 0.8 ** | 4.5 ± 1.0 * | 2.3 ± 0.2 * | | | $fcomb_{2.5}$ | 4.7 ± 1.1 * | 7.6 ± 0.7 * | -0.6 ± 1.5 * | | | $fcomb_{2.2}$ | 5.7 ± 1.9 * | 4.7 ± 3.1 * | -0.3 ± 1.8 * | | **Add** | $fcomb_{1}$ | 1.0 ± 0.3 * | 2.6 ± 0.3 * | - | | | $fcomb_{2.5}$ | 2.4 ± 0.5 * | 6.3 ± 0.2 * | - | | | $fcomb_{2.2}$ | 12.6 ± 0.2 ** | 6.0 ± 0.3 * | - |

Response to Official Review of Submission11182 by Reviewer TgMi

* **W1** The experiments show that ET signals can improve the accuracy of reward modeling. As the reward model is used for RLHF processes like PPO and Best-of-n sampling, I would suggest the authors add relevant experiments to show the improvement in this down-stream application of the reward model. We agree that it would be valuable to demonstrate the impact of the ET-enhanced reward model on downstream RLHF processes like PPO and Best-of-n sampling. Unfortunately, due to computational constraints, we were not able to include those additional experiments in this work. We hope the presented experiments effectively demonstrate the model's potential within our current resource limits and reserve the investigation of performance gains in these downstream applications in follow-up research. * **W2** What is the size of ET feature extractor? If the size of the ET component is not negligible, it might not be a fair comparison between the original model the model with ET feature extractor. Thank you for this feedback. To address your concerns about the model size comparison, we included the following information in the Appendix A.1.2: The first ET feature extractor has 17.5M parameters, while the second one has 125M parameters. More importantly, the ET component remains frozen during fine-tuning, so its parameters are not updated for the reward modeling task. We believe this makes the comparison with the baseline fair, as the overall model capacity is not significantly different. While the reward models we use are quite large (7-8 billion parameters), we agree that the relative size of the ET component is still worth considering. We have expanded our "Discussion" section to acknowledge this as a potential factor to keep in mind when interpreting the results.

Response to Official Review of Submission11182 by Reviewer sHLc

* **W1** Very minor, but I think section 2+5 could be better structured, perhaps by moving them both into section 2. I was a bit surprised on my first read to see a related work section near the end, since section 2 discusses much of the reward modeling literature; I think it makes sense to put these in the same spot, and I liked the placement near the beginning. Thank you for the helpful suggestion. We agree that moving section 5 after section 2 will improve the flow and clarify the structure. However, we believe keeping the "Related Work" separate from "Background" serves a purpose. In the revised version, we have applied this change, per your recommendation. * **W2** Because the datasets are somewhat constrained, it's not clear that this would provide consistent gains over existing reward models in a large-scale setting (could this be a useful signal only in small data regimes?). I don't think the paper necessarily needs to provide evidence against this concern, as this might require excessive amounts of compute, but I think more discussion of this possibility would help temper the claims in the work. You raise a valid point that the dataset constraints may limit the generalizability of the findings. While the paper does not provide definitive evidence against this concern, we have expanded the "Discussion" section to acknowledge this as a potential challenge and suggest directions for future research to explore the approach's performance in larger-scale settings. We hope this helps provide a more balanced perspective. * **Q1** You don't scale this approach to larger models/datasets (understandably!). Do you foresee any potential challenges of scaling this approach other than compute cost? Do you believe this data would be best used to fully replace existing methods for reward modeling, or in conjunction? Thank you for the insightful feedback. We do not foresee any major challenges in scaling our approach, beyond the computational cost. The use of gemerative models for producing ET features should allow the solution to scale to larger datasets and models. Moreover, we believe the ET data can enhance existing reward modeling methods, rather than fully replacing them. The goal is to incorporate this additional signal to better capture user preferences, while still leveraging the strengths of current techniques. We have expanded the "discussion" in the paper to clarify this perspective. * **Q2** Somewhat related to the above-- do you have a sense of any error category differences between RMs trained with ET data and without? The datasets we examined include both preferred and non-preferred annotations, typical of those used in human alignment tasks. Although we did not directly compare error categories between reward models trained with and without ET data, we believe that using data more closely aligned with the ET data used in the predictor model's training would enhance overall accuracy. In other words, we believe that preference data more similar to the data used for training the ET predictors could benefit from greater gains in performance. However, you raise a valid point and we believe that, as the fixation generation models and available datasets continue to evolve, the potential issues you highlighted can be further addressed.