RadEyeVideo: Enhancing general-domain Large Vision Language Model for chest X-ray analysis with video representations of eye gaze

Response to Weaknesses

> **W1** Methodological Clarity: a) The paper lacks sufficient detail about the video prompt integration process, particularly how the video information is encoded and processed by the LVLMs. The technical implementation of combining video sequences with textual prompts could be more thoroughly explained. b) There's limited discussion of potential alternatives to full video sequence processing that might achieve similar results with lower computational costs. We appreciate the reviewer’s interest in understanding the integration process for video prompts. As illustrated in Figures 1 and 2, we show how the visual (video) and textual components are designed to incorporate eye gaze information. In addition, Section 2 (RadEyeVideo) provides detailed descriptions of the encoding and processing of video sequences, including uniform sampling and how the eye-tracking data is transformed into a sequence of frames suitable for LVLM inputs. To address potential alternatives to full video sequence processing, we actually used filtering techniques to reduce noise and uniformly sampled 16 representative frames to balance computational efficiency with performance (detailed in Section 2). The ablation study on page 10 demonstrates how this sampling approach avoids the computational burden of processing full video sequences while maintaining diagnostic performance. *** > **W2** Data Collection and Generalizability Limitations: a) The methodology requires synchronized eye-tracking data collection during radiologist readings, which is resource-intensive and difficult to scale. b) The approach may not generalize well to scenarios where real-time eye-tracking data is unavailable. c) The current evaluation limited to a small dataset. We designed this study following extensive interviews with radiologists to ensure its relevance to clinical practice. The primary aim was to enhance general domain LVLMs for medical analysis by integrating human expertise through eye gaze video sequences. While collecting synchronized eye-tracking data can be resource-intensive, our approach demonstrates that incorporating this data improves model performance compared to medical LVLMs, even without additional training. This improvement offsets the costs associated with eye-tracking data collection by reducing the need for extensive model retraining. For scenarios where eye trackers are unavailable, our method can adapt by leveraging alternative approaches, such as mouse cursor movements as a proxy for eye gaze or human expertise. However, the findings from our study underscore the critical role of real eye gaze data, which offers imporved diagnostic value and supports the integration of human expertise into AI-driven workflows. We believe this work highlights the immense potential of eye gaze data to enhance diagnostic accuracy. Given the high stakes of medical diagnosis, even incremental improvements in model performance can lead to better patient outcomes and reduced risks associated with AI errors, such as hallucinations or incorrect responses. We understand the reviewer’s concern regarding generalizability and fully acknowledge the importance of testing on multiple datasets. In fact, we explicitly highlighted this as a limitation of our work in the paper. However, at the time of conducting this research, MIMIC-Eye was the only publicly available medical dataset with eye gaze information available. Despite this limitation, our proposed approach is inherently dataset-agnostic and can be seamlessly applied to other datasets as they become available. We also have plans to extend this work by exploring its applicability to additional modalities, starting with endoscopy, where eye gaze data can provide valuable insights. We believe these future directions will further enhance the generalizability and impact of our approach. *** > **W3** Computational Efficiency Concerns: The direct use of complete eye-tracking video sequences as prompts likely increases computational overhead significantly. There's limited analysis of the computational trade-offs or optimal sampling strategies. The method may be computationally prohibitive for real-time clinical applications. Once again, as pointed out in the response to **W1**, we don't use complete eye-tracking video sequences. Our sampling strategy was carefully designed to optimize efficiency with ablation study as shown on page 10, which shows even improved performance with 16 frames as shown in the Figure 4 on page 10. Also, we respectfully argue that computational efficiency is a key contribution of our work. Video-based general LVLMs enhanced with eye gaze data outperform task-specific medical LVLMs, without requiring computationally intensive training. The additional computational cost incurred by video prompts is minimal. The GPU memory usage increased by only ~10% (from 20.0GB to 21.9GB) when switching from eye gaze heatmaps to video sequences.

Response to Weaknesses

We really appreciate the time and effort you have dedicated to reviewing our manuscript! > **W1** Although this idea is interesting, it still relies on temporal and spatial information in the inference phase, which is difficult to apply to real clinical scenarios. Do the authors consider involving multiple information inputs only in the training phase and simulating zero-shot scenarios as much as possible in the inference phase? We respectfully clarify that our approach does not involve training the model entirely, which is highlighted throughout our paper. Instead, the multimodal data, including the spatial-temporal dynamics of eye gaze, are utilized during the inference phase to enhance the performance of general domain LVLMs on medical tasks. We designed this study following extensive interviews with radiologists to ensure its relevance to clinical practice. The primary aim was to enhance general domain LVLMs for medical analysis by integrating human expertise through eye gaze video sequences. While we acknowledge the challenge of real-world clinical scenarios, our method was deliberately structured to support radiologists in practice by leveraging eye gaze information to augment model performance without requiring extensive retraining for medical tasks. The findings from our study underscore the critical role of real eye gaze data, which offers unparalleled diagnostic value and supports the integration of human expertise into AI-driven workflows. We believe this work highlights the immense potential of eye gaze data to enhance diagnostic accuracy. While collecting synchronized eye-tracking data can be resource-intensive, our approach demonstrates that incorporating this data significantly improves model performance compared to task-specific LVLMs, even without additional training. This improvement offsets the costs associated with eye-tracking data collection by reducing the need for extensive model retraining. Given the high stakes of medical diagnosis, even incremental improvements in model performance can lead to better patient outcomes and reduced risks associated with AI errors, such as hallucinations or incorrect responses. *** > **W2** The study’s findings are limited by the small size of the MIMIC-Eye dataset, which may not fully capture the variability in real-world clinical settings, raising questions about the generalizability of the results. We understand that the size of the MIMIC-Eye dataset may limit the variability captured in this study. In fact, we explicitly highlighted this as a limitation of our work in the paper. However, MIMIC-Eye was the only publicly available medical dataset with eye gaze information at the time of research. Despite this limitation, our work demonstrates the potential of eye gaze information in improving LVLM capabilities for medical tasks, laying the groundwork for future research with larger and more diverse datasets. In fact, our proposed approach is inherently dataset-agnostic and can be seamlessly applied to other datasets as they become available. We also have plans to extend this work by exploring its applicability to additional modalities, starting with endoscopy, where eye gaze data can provide valuable insights. We believe these future directions will further enhance the generalizability and impact of our approach. *** > **W3** The comparisons primarily focus on selected models with minimal tuning for this domain. Including a wider range of task-specific medical LVLMs could provide a more comprehensive evaluation. We appreciate the suggestion and would like to highlight that we included CXRMate, a model specifically trained for chest X-ray report generation. Additionally, we evaluated our approach against CheXagent and CXR-LLaVA, two task-specific medical LVLMs trained for both report generation and diagnosis. These models were carefully chosen as strong baselines representing state-of-the-art approaches in the medical domain for both report generation and diagnosis. Expanding the scope to include additional models and tasks would be a valuable future direction as more advanced task-specific LVLMs become available. However, we believe our current comparisons provide a robust evaluation framework for the two mostly used tasks, report generation and diagnosis, demonstrating the effectiveness of our proposed method.

Response to Questions

> **Q1 Why the paper lacks an section of related work? e.g., some recent evaluation work of Med-LVLMs Although we did not designate a separate "Related Work" section, we included relevant recent works on Med-LVLMs, particularly those involving eye gaze information, in the introduction section (lines 50–60). These references provide context for our contributions and explain how our work differs from or builds upon prior studies. As the primary focus of our study is on incorporating eye gaze data as video input to enhance general LVLM capabilities for medical report generation and diagnosis, we believe the suggested evaluation works are not directly aligned with the goals of our research. *** > **Q2** The formats of reference is weird. e.g., in Line 311-321. Please check it. Thank you for pointing this out. We identified formatting issues where spaces were missing before some citations. We revised these errors with proper formatting.

Response to Weaknesses

Thank you for dedicating your time to review our paper! > **W1** In Figure 2, the authors illustrate different prompting methods; however, the figure does not clearly distinguish between the use of text descriptions and video inputs. This ambiguity makes it challenging to understand whether the authors used text descriptions to guide the prompt along with the video input or if they only provided video data. We respectfully clarify that Figure 2 exclusively focuses on the textual component of the prompt. For a clear depiction of how the visual prompt (video) input, specifically eye gaze data, is utilized, readers can refer to Figure 1, where our method, RadEyeVideo, is explicitly distinguished from preexisting prompts. Together, Figures 1 and 2 distinctly illustrate the integration of visual (video) and textual components in our approach. To further address the concern raised, we added a connecting sentence in Section 2.6 to explicitly link these two figures and clarify their complementary roles in illustrating our methodology for the revised version. We trust that these revisions will provide sufficient clarity and coherence for readers to fully understand the distinction and interplay between the textual and visual components of our prompts. *** > **W2** The author's experiments conducted on only one dataset are clearly insufficient in terms of persuasiveness. This limitation may affect the generalizability and applicability of the research findings. To enhance the credibility of the study, it is recommended that the authors validate their approach on different datasets. We understand the reviewer’s concern regarding generalizability and fully acknowledge the importance of testing on multiple datasets. In fact, we explicitly highlighted this as a limitation of our work in the paper. However, at the time of conducting this research, MIMIC-Eye was the only publicly available medical dataset with eye gaze information available. Despite this limitation, our proposed approach is inherently dataset-agnostic and can be seamlessly applied to other datasets as they become available. We also have plans to extend this work by exploring its applicability to additional modalities, starting with endoscopy, where eye gaze data can provide valuable insights. We believe these future directions will further enhance the generalizability and impact of our approach. *** > **W3** When the authors use radiologist's eye-tracking sequences as video input, this approach does enhance the model's understanding of prior knowledge in the diagnostic reading process to a certain extent. However, converting eye-tracking data into video format substantially increases the number of tokens, leading to significant computational resource consumption and longer inference times for the model. While we appreciate this observation, we respectfully clarify that converting eye-tracking data into video format does not result in a substantial increase in computational resource consumption. In fact, computational efficiency is one of the key contributions of our work. Our results demonstrate that video-based general LVLMs outperform specialized medical LVLMs, showcasing the potential to bypass the extensive computational resources typically required for model training in medical domain tasks. To provide further clarification, the increase in GPU memory usage when incorporating eye gaze data as video, rather than as a heatmap, is marginal—approximately 2GB, from 20,076MB (eye gaze heatmap) to 21,954MB (eye gaze video) using the LLaVA-OneVision model. This represents only a 10% increase in GPU memory usage, which is minimal compared to the resource demands of training specialized medical LVLMs, often requiring at least double the computational resources. We have elaborated on this point in the revised version to better emphasize the efficiency and practicality of our approach, particularly in leveraging video inputs with manageable computational overhead.