Which Experiences Are Influential for RL Agents? Efficiently Estimating The Influence of Experiences

Response to Reviewer A595

Thank you for your valuable comments. We will revise the paper based on your feedback. **Q1.** the theoretical justification is lacking because the assumptions (1 and 2) don't seem realistic. Basically, the authors assume that the masks enforce some sort of leave-out rule. It makes sense since they try to minimize overlap (App B), but it is the assumptions are unjustified. The authors should provide empirical evidence supporting assumptions 1 and 2, and discuss the implications if these assumptions don't fully hold in practice. **A1.** For Assumption 2, we ensure in our implementation that there is no overlap between the masks $m_i$ and the flip masks $w_i$. Therefore, we believe this assumption holds in practice. On the other hand, Assumption 1 may not be fully satisfied in our current implementation. If Assumption 1 is not strictly met, it means that applying the flip mask corresponding to a specific experience may not entirely exclude the influence of that experience. To fully satisfy Assumption 1, one potential implementation would involve completely eliminating any overlap between the masks $m_i$ and $m_j$ ($i \neq j $). However, this approach would require ignoring interactions between experience groups, which is impractical. For this reason, we did not adopt such a method in our implementation. In summary, there is a trade-off between strictly meeting Assumption 1 and maintaining practicality. Research and development on methods and implementations that balance these factors remain future work. We will add a discussion of this point in Appendix I. **Q2.** missing analysis of results, mostly why the results in figures 3, 4 and 6 look the way they do. The authors should explain why the plots for some environments show lower ratios / performance than other, the reasons for instability in some cases (including shaded confidence intervals). Figure 4 is explained but this explanation is unclear. Specifically, it is unclear why this figure suggests that the agent is overfitting to older experiences. **A2.** Regarding the explanation of Figure 4, the phrase "overfitting to older experiences" may have been an overstatement. We have revised the manuscript to clarify that self-influence is concentrated on older experiences, without implying overfitting. For the other points (Figures 3 and 6) you mentioned, we will include additional explanations and analyses in the revised manuscript. **Q3.** figure 4 heatmap scales are all different, making it very difficult to read and compare the graphs. Either use consistent scales for all plots or at least keep colors consistent regardless of the scale, i.e., if 1 is yellow and -1 is blue inn one scale, then in another scale -1 is still blue, and -4 can be a new color, e.g., green. **A3.** We are currently revising the figure to address this issue. **Q4.** does not consider what happens if the buffer is full, something that will eventually happen if training persists. **A4.** If computational resources allow, we will conduct experiments to address this scenario. \# In PIToD, even if the buffer becomes full and older experiences are overwritten by newer ones, this does not pose a fundamental problem. However, if we want to estimate the influence of the overwritten experiences later, it is necessary to record the corresponding masks (or the random seeds used to generate those masks). **Q5.** no comparison to other methods. E.g., PER is definitely comparable. **A5.** We are currently conducting experiments with SAC. Once those are complete, we will add experiments with SAC+PER as well. **Q6.** Could I use PIToD and PER together? **A6.** PER performs weighted sampling of experiences based on TD errors evaluated with the Q-function ([1], Algorithm 1, line 11). If it is acceptable to ignore the influence of individual experiences on the sampling process, then PIToD and PER can be used together. In such a case, combining the two methods would involve replacing PIToD's uniform sampling scheme with PER's prioritized sampling scheme. [1] https://arxiv.org/pdf/1511.05952 **Q7.** Your algorithm as many loops. Can these ToD iterations be efficiently batched? **A7.** For line 8 of Algorithm 2, when using $L_{pe, i} $, $L_{pi, i}$, and $L_{bias, i}$, the operations are performed in batches. These batch operations are implemented in the provided code under `redq/utils/bias_utils.py` in the supplemental material. $L_{ret}$ involves interaction with the Mujoco (CPU) environment for return calculation, so batching is not applied in this case.

Response to Reviewer YodU

Thank you for your valuable comments. We will revise the paper based on your feedback. **Q1.** More explanation in Section 4 of the PIToD method would be very helpful for a reader's understanding. "Thus, some readers may suspect that the parameters dropped out by (i.e., the parameters obtained by applying ) are not influenced by ." - This sentence caused a lot of confusion to me when reading the paper. The phrasing seems to suggest that the parameters dropped out are indeed affected, but this doesn't seem to be the case (as the reader would suspect). **A1.** We intended to use "suspect" in the sense of "think" or "believe," rather than "doubt," which may have caused some confusion. The parameters dropped out by the mask are, in theory, not influenced by the corresponding experience. We have revised the sentence in Section 4 to ensure that this point is communicated clearly and unambiguously. Thank you for highlighting this potential misunderstanding. **Q2.** I am also struggling to understand Appendix Section A.... Looking at the equality, doesn't it mean that the gradient for $i'$ is 0 for everything but $i'$ due to the indicator? **A2.** > Assumption 1 seems extremely strong and appears to be almost exactly defining the property you are looking to prove. Assumption 1 may be a strong assumption that is difficult to reconcile with practical implementations. As mentioned in response A1 to Reviewer A595, strictly satisfying this assumption in practical implementations is not easy at this stage. i.e., Prioritizing practicality means that fully meeting this assumption remains challenging, and resolving this gap is a topic for future work. Assumption 1 essentially assumes that the Q-function and policy function using $w_i'$ can be replaced by functions dominantly influenced by the experience $e_i'$. On the other hand, what we aim to prove is that, when training under the PIToD framework, applying the flip mask $w_i'$ corresponding to $e_i$ enables us to isolate parameters unaffected by $e_i$. > Looking at the equality, doesn't it mean that the gradient for is 0 for everything but due to the indicator? Yes, that is correct. **Q3.** Section C in the appendix contains a lot of interesting content. The findings from the Group Mask preliminary experiments in the appendix seem quite important - "In our preliminary experiments, we found that the influence of a single experience on performance was negligibly small". This should be mentioned at the very least in the main paper. "Key implementation decisions to improve learning." is also very important. The implementation details, and the architectural experiments you conducted should not be relegated to the appendix in this manner. Especially since Figure 10 shows they are critical to your method working. **A3.** As per your suggestion, we plan to include several key insights and findings in the main paper. **Q4.** What is the importance/significance of the differences in the ratios between policy eval/improvement? For Eqn 8 I can understand that the td error if higher for an experience you haven't trained on, but for Eqn 9 I do not quite see why the action chosen by $\pi_{\theta, w_i}$ should have a lower estimated Q. **A4.** $\pi_{\theta, m_i}$ is trained to maximize the estimated Q (c.f. Algorithm 2, line 6), whereas $\pi_{\theta, w_i}$ is not trained in the same way. As a result, $\pi_{\theta, w_i}$ is expected to have a lower estimated Q. **Q5.** Section 5.2 is very misleading since you're estimating the time it would take LOO. Given that important sections that have been pushed into the appendix, I would advise replacing these with other more relevant/concrete content. **A5.** Thank you for pointing this out. To address your concern, we are considering replacing this section with other (e.g., Appendix C). Additionally, we will explicitly clarify in the text that the LOO time presented in this section is based on an estimated calculation. **Q6.** "In our setup, $L_{ret}$ is estimated using Monte Carlo returns collected by rolling out policies" - How many rollouts are used for each estimation? Additionally, how do you utilise Eq. 10 to identify experiences? Are you rolling out the agent multiple times to collect MC returns? Is this factored into your training budget and reflected in Figure 6? **A6.** For each group of experiences included in the replay buffer, the agent is rolled out for 10 test episodes to estimate the MC return. Using these estimates, Eq. 10 is calculated to estimate the influence of each experience group. The experience group corresponding to the flip mask with the highest Eq. 10 value is identified as having a negative influence. The additional budget required for influence estimation is not reflected in Figure 6 but is shown in Figure 13.

Response to Reviewer cTe3

Thank you for your valuable comments. We will revise the paper based on your feedback. **Q1.** The metric used to show self-influence appears inappropriate. As noted in L246, is in a state where it has not been trained on , so is expected to be greater than zero in most cases, regardless of whether is beneficial for learning. Additionally, since , is likely always less than zero. Thus, these metrics do not directly indicate whether the experience has a positive or negative influence on RL learning. **Q1'.** [About Weakness 1] You mentioned that the PI and PE metrics indicate whether a specific experience has a positive or negative effect. Could you clarify this further? The current metrics seem to only reflect whether or not learning utilized $e_i$. **A1.** Regarding the PI and PE metrics in Section 5.1, we did not claim that "the PI and PE metrics indicate whether a specific experience has a positive or negative effect." It seems there might be a difference in interpretation of the terms "positive" and "negative." In Section 5.1, we use "positive" to mean a value greater than or equal to zero and "negative" to mean a value less than or equal to zero. We are not using "positive" to imply that an experience is beneficial for learning or "negative" to imply that it is harmful. **Q2.** [About Weakness 2] Is this approach fundamentally different from simply creating a policy ensemble through dropout and selecting the best-performing one? If the truly optimal is selected, does it necessarily mean that the experience has a negative impact? **A2.** This approach fundamentally differs from a simple policy ensemble in that specific policies within the ensemble are trained exclusively on specific sets of experiences. If the optimal policy has not been trained on a particular set of experiences and outperforms those that have, it indicates that the specific set of experiences has a negative impact. **Q3.** [About Weakness 3] Could you summarize the experiments in the appendix and explain what each aims to demonstrate? It would be more beneficial if these results were integrated into the main paper. **A3.** We will add a summary of the objectives and results of the experiments in the appendix to the main paper. **Q4.** [About Weakness 4] Could you also present the experiments mentioned above? **A4.** We are currently conducting comparison experiments with SAC, and we will add the results once they are complete.

Response to Reviewer xR4M

Thank you for your valuable comments. We will revise the paper based on your feedback. **Q1.** The paper do not have a theoretical underpinning for their approach though some reader may find the idea intuitive, that said, personally, I'm not a fan of removing samples from experience buffers; **A1.** We provide theoretical analyses in Appendix A to justify the use of PIToD. **Q2.** because the way I see it, deleting "negatively influential experiences" seems to be a lenient/hysteresis update commonly seen in optimistic approaches, it may be useful in same case but should be used with caution since it may hinder learning and cause bias; a comparison with similar approaches is not included. I think instead of removing samples, the safer approach is to use gradient clipping or learning rate schedules to stabilize training. Techniques like gradient clipping cap the maximum value of gradients to prevent instability without losing any data -- **A2.** Our method (PIToD) is primarily designed to identify and remove experiences that hinder learning or introduce bias. Additionally, the main objective of our work is not to stabilize training but to estimate the influence of experiences. In the context of online reinforcement learning, our study is the first attempt to estimate the influence of experiences and, to the best of our knowledge, no similar studies (approaches) exist. **Q3.** Data removal in RL may massively hinder exploration, especially in non-linear environments, speaking of which -- **A3.** In our experiments, data removal is not performed during training (c.f. Algorithm 4). Therefore, in our experiments, data removal does not hinder exploration. **Q4.** the paper only evaluate on classic mujoco, and lacks variety of evaluation task, speaking of evaluation -- **A4.** In our paper, we conduct experiments on 4 Mujoco tasks and 8 DMC tasks (Section 6 and Appendix G). The number of tasks would be comparable to previous works accepted at ICLR (e.g., [1]). [1] https://openreview.net/forum?id=PczQtTsTIX **Q5.** The evaluation also lacks other sota methods for comparison, since the promise was to have better performance, rather than better theoretical understanding, which goes back to my first point. **A5.** We provide theoretical analyses (c.f. A1). Additionally, since our work is the first to address this type of study in online RL, there are no existing SOTA methods for comparison (c.f. A2). **Q6.** How does sample rejection based on gradient relate to hessian? **A6.** We may not fully understand your question. Are you asking whether our method (PIToD) relates to existing research (e.g., [2]), which uses Hessian-based approaches for influence estimation in a supervised learning setting? [2] https://arxiv.org/pdf/1703.04730