Vision-Language Navigation with Energy-Based Policy

Summary

To attempt to solve the catastrophic failure issues that arise from behavioral cloning, the authors introduce Energy-based Navigation Policy (Enp) which uses an energy-based model to learn the expert policy from demonstrations. The authors claim that by doing so Enp prioritizes learning the full trajectory instead of single-time step decisions in BC.

Strengths

**S1.** The issue of catastrophic failure from BC is an important one that which is likely applicable to the setting of VLN — this justifies the importance of the problem. **S2.** The method is simple and described appropriately, both of which are strengths of the work.

Weaknesses

**W1.** *Overall improvements are low.* The performance improvements provided by ENP are small compared to the BC baselines, often around $1-2\%$ in terms of SR and similar in SPL. This is particularly poignant when compared to the improvements provided by semantic grid maps. By itself, this would not necessarily be a major issue, if it was clear where ENP was helping (W2). **W2.** *It is unclear when ENP is better than BC, and if the intuitive justification of catastrophic failure applies.* A more in-depth analysis of the success/failure cases of using ENP vs BC is warranted. Metrics like SR suggest that ENP can be helpful in avoiding the failure that BC can induce, but it is not entirely clear this is the case — when is ENP helping? Given the justification of the catastrophic failure of BC, you would expect this would be in longer trajectories, but the example in Figure 3a does not justify that given its short trajectory length. Could you create a benchmark specifically to test this? Including this analysis could boost the case for the motivation behind ENP significantly. **W3.** *ENP is likely more expensive to train than BC.* The authors should include a compute comparison for the training of ENP vs BC. While the inference runtime is the same, training time for ENP will likely be significantly influenced by $I$. The combination of W1 and W2 severely hinder the practical motivation of the method. Minor comments: - It’s not really required to include all baselines in the main results tables. E.g., Seq2Seq, SF and RCM are pretty outdated at this point and always beaten by other methods — it’s better to focus the comparison with the most recent methods and leave the full tables to the appendix.

Questions

**Q1.** Why are the DUET or, e.g., BEVBert/GridMM, paths not shown in Figure 3b too? If multiple agents face the same issue, that would show this is the problem of observability. **Q2.** What are the $\pm$ values in Table 3, is it the standard deviation? If so, why is it not reported for the other methods? **Q3.** Do the authors have any intuition on why increasing $I_{\text{pre}}$ and $I_{\text{ft}}$ could result in a lower SR/SPL (Table 7)?

Rating

5

Confidence

3

Soundness

2

Presentation

3

Contribution

2

Limitations

The authors do not discuss the limitations of their method. The limitations paragraph in Appendix A.2. does not reflect the particular ENP method they propose and is instead generic to the problem. Training time is one potential limitation that should be studied and discussed in further detail.

Summary

This paper studies vision-and-language navigation (VLN) by addressing the problems of error accumulation in the Markov decision process and the difficulties in designing an optimal reward function, respectively, in the commonly applied behavioral cloning (BC) and reinforcement learning (RL) approaches. It proposes an energy-based policy (ENP) for training the agent by estimating the unnormalized probability density of the expert’s occupancy measure, optimizing the entire trajectories rather than the single-time step decisions in BC. Meanwhile, to facilitate the MCMC sampling in ENP, a Marginal State Memory is introduced for the agent as an online reply buffer. The proposed method has been evaluated on multiple popular VLN agents and several benchmarking datasets and environments, together with detailed analyses of the ENP and design choices.

Strengths

1. The energy-based navigation policy proposed in this paper introduces a new training perspective for VLN and potentially for broader visual navigation problems. The idea is very well-motivated as it addresses the limitations of commonly applied BC and RL approaches. As a researcher in visual navigation, I am happy to see the exploration of such a fundamentally different training approach and the fact that the method works well across models and settings. 2. Arguments and design choices made in this paper are well-justified by analyses and experiments (e.g., Section 3.3 and Section 4.3). 3. The proposed idea has been evaluated across various popular VLN baselines (e.g., EnvDrop, VLN-BERT, and DUET), different datasets (R2R, REVERIE, and RxR), and different environments (e.g., discrete and continuous), achieving obvious improvements and showing the effectiveness of the method. 4. Overall, the paper is nicely written. Contents, tables, and figures are clearly presented.

Weaknesses

1. The proposed ENP has only been evaluated on a limited scale of VLN datasets and models. It is unclear if the method will still benefit larger models, or the case where more sufficient training data is available to learn a very generalizable policy through IL/BC (e.g., 4.9M ScaleVLN data [49] vs. the original 14K R2R data). I am concerned that IL/BC is much more friendly and stable than ENP to scale with data. 2. There is no comprehensive successful/failure case analysis of ENP versus existing training methods. It is unclear exactly in what scenarios an agent trained with ENP shows an advantage. I need to mention that I have very little knowledge about energy-based models, so I might have missed some key details in mathematics. I will rely more (and probably adjust my review based) on the comments from the other reviewers on the theoretical part of this paper.

Questions

No other questions; I wish the authors could kindly respond to my concerns in Weaknesses.

Rating

7

Confidence

2

Soundness

3

Presentation

4

Contribution

3

Limitations

The authors have discussed some limitations of the proposed method and its potential societal impact. As for suggestions, mainly the two points in Weaknesses – test on large-scale training data and run thorough result analysis.

Summary

This paper proposes to learn a better Vision-and-Language Navigation agent by jointly optimizing the action and the state. Specifically, they propose an energy-based expert policy, where they adds an explicit objective for navigation exploration and incorporates prior knowledge from the expert demonstrations. Empirical results on multiple VLN benchmarks (e.g., R2R, REVERIE, R2R-CE) demonstrates the effectiveness of the proposed approach.

Strengths

1. The proposed approach is supported with good theoretical foundations, and this paper provides a detailed derivation of the objective, as well as providing explanation on how to place the approach in previous literature with imitation learning approaches. 2. Empirical results on multiple VLN benchmarks demonstrate the effectiveness of the approach, and also demonstrates this approach's generalization to different base VLN agents. 3. Detailed ablations to investigate different components of the approach (e.g., step size, gaussian noise, SGLD loop).

Weaknesses

1. The proposed approaches work pretty well on sota agents like DUET, but it's underinvestigated whether the proposed approach also benefits VLN agents when having large-scale data. Specifically, it's important to have experiments on ScaleVLN or HM3D-AutoVLN. 2. The approach might be sensitive to hyperparameters as suggested in Table 6 and Table 7, which might need additional efforts in hyperparameter tunning when generalized to different environments/agents/tasks.

Questions

Please refer to weakness.

Rating

6

Confidence

5

Soundness

3

Presentation

3

Contribution

3

Limitations

N/A

Final Rating

Thanks for addressing my concerns. The additional experiments are very strong. I will raise my rating to weak accept (6).