Responses to the questions
**Q1: Why call it “destroy” since the original trajectory isn’t always discarded? A more intuitive name could be “backtrack,” “rewind,” or anything that doesn’t suggest destruction.**
We are in agreement on this matter. We have updated our manuscript to incorporate the term "backtrack" instead of "destroy."
---
**Q2: The top plots in Figure 5 are strange: it looks like some curves go beyond 100% accuracy. Could you either fix them so we can still see the curves on the plot or explain what is happening?**
Please note that Figure 5 is plotted accurately in accordance with the baseline from the paper by Shen et al., 2023, where we have set our accuracy metrics following their guidelines:
$\text{Acc}(p(x;\theta)) = 100 \times \text{min}(\frac{E_{p(x;\theta)}[R(x)]}{E_{p^{*}(x)}[R(x)]},1)$
This provides an upper bound of 100% accuracy. In response to your suggestion, we have included a graph illustrating accuracy beyond the 100% mark in Appendix B.7.
---
**Q3: How can LS-GFNs recover from a biased backward policy? In other words, assume the forward policy is fine but the backward policy always backtracks to states that yield the same (high reward) candidate objects — how can LS-GFNs overcome this lack of exploration?**
Biased backward policies can indeed pose challenges. In such scenarios, we can employ an $\epsilon$-greedy method to introduce exploratory back-sampling into the biased backward policy. It's worth noting that LS-GFN performs exceptionally well in MaxEnt GFN, even when the backward policy is set to a uniform distribution (i.e., $\epsilon = 1$).
Building upon your valuable feedback, one potential avenue for future research could involve fine-tuning the backward policy to maximize the local search acceptance rate. This approach would aim to ensure that the backward policy identifies diverse intermediate states that lead to improved rewards without exhibiting a bias towards yielding the same candidate objects repeatedly. We appreciate your insightful comment.
---
**Q4: Please confirm that lines 7 and 8 in Algorithm 1 aren’t swapped. If they aren’t (which partially addresses my question above), wouldn’t swapping them and extending them further improve exploitation? Maybe this should be added as an ablation as well.**
No swapping is involved here. We update every sample to the dataset, irrespective of whether it is accepted or not. The rationale behind this approach is that every sample where the reward is calculated is considered valuable. During the refinement phase, our primary objective is to explore the local region in order to discover high-reward samples rather than determining whether the found sample should be used in training.
In the training process, we employ PRT to sample trajectories in proportion to their rewards from the dataset. Consequently, accepted samples have a higher likelihood of being included during training.
Regarding your suggestion to swap lines 7 and 8, it would entail updating the data only with accepted samples. However, as you pointed out, while this could enhance exploitation, it might excessively hinder exploration. To shed more light on this, here are the new results of our ablation study:
| Column 1 | Number of modes| TopK reward |
| -------- | -------- | -------- |
| Swapped version | 754 ± 6 | 0.66 ± 0.00 |
| Original LS-GFN | 793 ± 4 | 0.67 ± 0.00 |
Based on the above results, the original LS-GFN gives higher performances than the swapped version.
---
**References**
Max W Shen, Emmanuel Bengio, Ehsan Hajiramezanali, Andreas Loukas, Kyunghyun Cho, and
Tommaso Biancalani. Towards understanding and improving GFlowNet training. In International
Conference on Machine Learning, pp. 30956–30975. PMLR, 2023