Weaknesses
- The paper does not contextualize its contributions correctly by failing to include a discussion on Process Reward Models~(PRM) that have already been proposed and leveraged in multi-step reasoning tasks [References 1-5]. The techniques proposed appear to be an application of these findings on the GUI Navigation domain, while I’m personally of the opinion that applications of techniques on new domains can still present novel insights beneficial to the community, I think the paper in its current state does not do that by glossing over specific details (see Q1-5) and not providing enough domain-specific discussion. I feel the paper will benefit from revealing more domain specific details: What are the nature of these tasks selected for testing the method? How long are typical interactions in dynamic tasks selected? How does the method fare on different tasks – are there specific characteristics of tasks where the current models perform poorly?
- The contribution statement (Section 7) claims more robustness and efficiency in complex GUI navigation but these notions are not defined in the paper anywhere. Improved success rate does not imply efficiency or robustness but these can be interesting to demonstrate by presenting alternate metrics such as the number of steps to goal (efficiency), repeated successful completion under disturbances such as ads in shopping tasks (robustness).
_References_:
[1] Lightman, H., Kosaraju, V., Burda, Y., Edwards, H., Baker, B., Lee, T., Leike, J., Schulman, J., Sutskever, I. and Cobbe, K., 2023. Let's verify step by step. arXiv preprint arXiv:2305.20050.
[2] Li, Y., Lin, Z., Zhang, S., Fu, Q., Chen, B., Lou, J.G. and Chen, W., 2023, July. Making language models better reasoners with step-aware verifier. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 5315-5333).
[3] Ma, Q., Zhou, H., Liu, T., Yuan, J., Liu, P., You, Y. and Yang, H., 2023. Let's reward step by step: Step-Level reward model as the Navigators for Reasoning. arXiv preprint arXiv:2310.10080.
[4] Wang, P., Li, L., Shao, Z., Xu, R., Dai, D., Li, Y., Chen, D., Wu, Y. and Sui, Z., 2024, August. Math-shepherd: Verify and reinforce LLMs step-by-step without human annotations. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 9426-9439).
[5] Snell, C., Lee, J., Xu, K. and Kumar, A., 2024. Scaling LLM test-time compute optimally can be more effective than scaling model parameters. arXiv preprint arXiv:2408.03314.
Questions
(Q1) Reward model training (Section 3.1): The annotation process provided in Appendix 8.1 is unclear for self-play trajectories – Is the agent initialized at specific contexts ($S_{t-1}, A_{t-1}, s_t$) in the AitW dataset to then score different actions based on ground-truth action? (Doesn’t this mean we just have more labels for the correct action at each context but not at arbitrary contexts). If not, how are you effectively mapping arbitrary contexts an agent might reach from the reset distribution of a task to the ground-truth labels in a static dataset? Can you provide more details for how much self-play augmentation was performed for training the process reward models considered in the experiment.
(Q2) History compression (Section 3.1): How much of the state interactions were compressed for evaluation? Do you compress the states at every decision making step or do you reuse previous compressed history? Am I right in assuming that the history compression is zero-shot GPT-4o (Appendix 8.3 points to a prompt but does not tell which model does this).
- While the prompt explicitly states do not speculate future actions, Figure 3 of the paper indicates that history includes “clicked the search bar to find flights”, which the Top-1 answer seems to assume to be the case (and interestingly ignores the fact that no keyword is visible in the screenshot for it to type onto).
- Can you comment on the general quality of zero-shot history summarization and the typical lengths of the states compressed by a VLM?
(Q3) Reflection Thought Generation (Section 3.3): What is the prompt that generates the reflection though for retry? Does it use the full uncompressed context in dynamic tasks? I think key domain specific details for these elements are missing in the paper.
(Q4) On TopK baseline (4.1): The paper states: “We simplify the process by selecting the most probable one. This ensures that the model still considers multiple possibilities but prioritizes highest probability action for execution” – This statement does not make sense to me, can the authors describe how sampling multiple actions enables this baseline to consider multiple possibilities?
(Q5) For Table 1. Could you comment on the stochasticity of action recommendation of the GPT-4o policy – should the evaluation be performed over multiple seeds for actual Top-K style approaches (like the proposed GuideNav)? In Table 3. Is VLM-FT300 row the GuidNav method that is fine-tuned with 300 tasks from Table 1 – if so, these do suggest some variance in performance for static assessment.