Cross-Domain Policy Adaptation via Value-Guided Data Filtering

Summary

The authors propose a method that does domain adaptation for RL. Unlike other approaches that do this via domain randomization, offline interaction from the target env, or adjusting the sim with system ID from a small real dataset, the aim is to learn from a large source of sim transitions and a small number of **online** interactions. The closest comparison to the work is DARC, which was also evaluated in a similar setting. DARC is from a class of methods that learns a reward correction term $\Delta r$ that captures the change in environment dynamics between source and target domain, and estimates $\Delta r$ using classifiers. This work suggests that DARC's estimation method is too conservative. In particular, given two tuples $(s,a,s_1')$ and $(s,a,s_2')$, if $V^\pi_{tar}(s_1') \approx V^\pi_{tar}(s_2')$, then for the purposes of value estimation $s_1'$ and $s_2'$ can be considered similar, even if the individual states themselves are quite different. This is an example of a case where DARC could produce a larger reward penalty. This work aims to estimate $V^\pi_{tar}(s)$ from a small number of interactions in the target domain + many interactions in the source domain. To do so, an ensemble of dynamics models $T_\phi(s'|s,a)$ is learned from target data. Sampling target state $s_{tar}'$ from the $T_\phi(s'|s,a)$ family gives an ensemble of Q-value estimates $Q_{tar}(s,a)$, and this lets us model $V^\pi_{tar}(s)$ as a random variable with some amount of uncertainty. The function is then updated with all target data, and some source data, where the source data is used only if it is measured as sufficiently "in-distribution" to the target environment. (In practice the authors use the top 25% most "in-distribution" source data.)

Strengths

I found the theoretical arguments reasonably clear, modulo a few typos (see later section), and on double-checking them, everything looks correct. I think the core idea makes sense: that for the purposes of learning, if you assume your value estimate $V$ is accurate enough, then it is sufficient to compare environments based on your value estimate $V(s_{src})$ vs $V(s_{tar})$ since the value function encapsulates all future variation. This idea is compelling.

Weaknesses

In terms of references, there are some approaches that use value function equivalence as an objective for learning their manipulable sim (i.e. RL-CycleGAN, if you view learning an image-to-image GAN as a manipulable sim). My more specific reservation is with the comparison to DARC, the most competitive of the baselines. The GridWorld example given shows an environment where DARC is overly pessimistic in bottleneck states, and fails to explore outside of that region. DARC has theoretical guarantees of correctness, so what is going on here? Perhaps in practice, the theory is too conservative (the field has seen similar things with TRPO / PPO using less-conservative approximations of theory). In this scenario, my natural response would be to not to derive a new method, but to try weighting the reward correction term $\Delta r$ in DARC by some hyperparameter. It is not very theoretically principled but would probably handle the pessimism alright. Does this approach meaningfully improve on DARC if the conservativism term was weighted? It's unclear. Setting this aside for now, if $V$ is learned well, then this paper (VGDF) makes sense. But to me this seems like a very strong "if" to make. It's a little chicken and egg, where $V$ should only be learned well if we did not need the source data in the first place? To determine the closest section of source data, we must already learn a 1-step dynamics model of the target environment $T(s_{t+1}|s,a)$, and estimate $V_{tar}(s')$ to estimate the future value from $s'$. If both those pre-requisites are true, it feels like we ought to just use a model-based method instead. As implemented, the paper always uses the 25% closest segment of the source data, but there is no guarantee this 25% closest segment is actually that close to the target data. The ablations in Figure 6 suggest that the separation function is doing something, but I am a bit concerned that performance does not seem to differ much for many different thresholds $\xi$. I am also not so fond of the exploration policy $\pi_E$. It makes sense but I believe the baselines do not use a separately trained exploration policy (please correct me on this if wrong). It doesn't seem like $\pi_E$ is a major component based on the ablation run, so this is a lesser concern for me.

Questions

Are there any results suggesting the accuracy of the value function $V$ during training? Or at least suggesting that any inaccuracies in $V$ are correlated during training? I would also appreciate more details on how the Gaussian dynamics models $T_\phi$ were learned - given that the paper's central argument is to measure state distance by $V_{tar}(s)$, the method section for how $V_{tar}$ is learned seems oddly rushed. Misc questions: line 580 of appendix: it should be $W_{j+1} = ... + V(s_{j+1})$ instead of $V(s_j)$ right? line 565 of appendix: I'm not sure why the proof is 4 lines. Lemma C.1 gives $\eta_1(\pi) - \eta_2(\pi) = \frac{\gamma}{1-\gamma} E[ E_{P_1}[V_{tar}(s')] - E_{P_2}[V_{tar}(s')] ]$. The end of Theorem B.2 is equivalent to, $\eta_1(\pi) - \eta_2(\pi) \le$ "the outer expectation, if you take the absolute value of the insides before integrating". So I'm not sure why there's a step where the $P_{src}$ and $P_{tar}$ terms are shuffled around before taking the absolute value, seems like needless notation unless I'm missing something.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Limitations seems addressed fine.

Summary

The paper is concerned with the online dynamics adaptation problem, where an agent is tasked to generalize from a source domain with cheap access to a target domain with a limited online interaction budget. The approach filters what data from the source domain is used in the target domain based on whether the state-action pair will result in a state of similar value in both source and target domain. This approach is theoretically justified and motivated in a toy example. The authors show empirical results in modified versions of the Deepmind control suite.

Strengths

The paper benefits from a relatively tight relation between theoretical motivation and practical implementation. The "value discrepancy perspective" is original in this context, and the authors carefully compare and ablate their method. The paper is easy to read and to understand. As sim2real transfer is a very real and pressing issue in robotics and control, I think the paper makes a significant contribution.

Weaknesses

The paper could potentially benefit from additional experiments in environments that are not in the "short-term control" regime that is typical for the Deepmind control suite, but more long-horizion like physical manipulation scenarios or a more complex version of the maze that is used for the motivating example. Since in such scenarios, value assignment is generally harder to do correctly, it might be interesting to see how the performance of value-based data sharing is affected in such a case.

Questions

- I did not really understand why the fact that DARC learns overly pessimistic value estimations should be (as argued in section 4.1 and elsewhere) a direct consequence of DARC being dynamics-based rather than value-based? Would it not be possible to create a "more forgiving" version of DARC that, for example, uses a data-filtering mechanism rather than the (arguably over-pessimistic) reward shaping mechanism it currently uses? If I did not misunderstand the motivating example, I feel like it does not really disentangle the two aspects of (1) reward shaping vs. data filtering and (2) dynamics-based vs. value-based, but then the difference in results is mostly assigned to (2). The argument that value difference is more long-horizon than dynamics difference makes sense to me, though.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

4 excellent

Contribution

3 good

Limitations

The limitations section is quite brief. I appreciate that space is very constrained, but maybe it would improve the paper to expand this a little for the camera-ready version.

Summary

This paper considers a setting of online dynamics adaptation, where the goal is to train a near-optimal policy in the target domain using transition data from the source domain and the target domain with different dynamics. The authors propose to select source domain data to train Q-functions if the value discrepancy between the source and the target domain is minimal. The authors propose Fictitious Value Proximity (FVP) that represents the likelihood of the source domain state value and select source domain transitions with the top 25% quantile of FVP. The authors evaluated the proposed method on environments with different dynamics, including kinematics and morphology change.

Strengths

1. This paper tackles the problem of generalizing policies across dynamics mismatch, which is significant in reinforcement learning. 2. The writing and the clarity are good. Also, this paper includes a theoretical performance bound controlled by value difference to support the claim.

Weaknesses

1. The proposed idea seems to be simple and obvious. VGDF wants to select source domain data with similar state transition dynamics for additional training data, and it uses a value function to measure the similarity of state transition dynamics. 2. Although VGDF demonstrates superior performance baselines in some environments, it has comparable or inferior results to a fine-tuning method on the target domain in other environments.

Questions

1. There seems to be a discrepancy between the method described in the paper and the code implementation of it. In the paper, VGDF selects the source domain samples based on the likelihood of the next state value $V(s’)$. On the other hand, in the code, VGDF selects the source domain samples based on the likelihood of $r + \gamma V(s’)$, which approximates the current state value $V(s)$. In the code, the ensemble of dynamics models predicts the reward $r$ and the next state $s’$ in the target domain (line 286 in vgdf.py). These predictions are then used to compute Fictitious Value Ensemble (FVE) of $r + \gamma V(s’)$ (lines 244 and 303 in vgdf.py) and Fictitious Value Proximity (FVP) of $r + \gamma V(s’)$ on source domain samples. (line 261 in vgdf.py). This means that VGDF chooses source domain samples based on $V(s)$ instead of $V(s’)$, which contradicts the description in the paper. It is unclear why the authors chose to use $r + \gamma V(s’)$ instead of $V(s’)$ in the code implementation. 2. According to Fig. 7, using $\pi_E$ seems insignificant. How exactly the exploration policy behaves in the source domain? I wonder whether maximizing $Q_{UB} = \max\{Q1, Q2\}$ actually leads to optimistic exploration.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

The authors included the limitations in the Conclusion section.

Summary

Disclaimer: I found it challenging to fully comprehend the paper. This may be due to either inadequate clarity in the writing or my own limitations in understanding the subject matter. This work introduces a method called Value-Guided Data Filtering (VGDF), which aims to enable online dynamic adaptation. By utilizing a set of state-action pairs from a source domain, VGDF selects relevant transitions to train a policy that performs well in a target domain with differing dynamics, all while not requiring the need for extensive interactions with the target domain. To assess the effectiveness of the proposed method, experiments are conducted on four Gym Mujoco environments that feature diverse, dynamic shifts, such as kinematic changes and morphological variations.

Strengths

- The main results depicted in Figure 4 exhibit considerable strength and indicate that VGDF outperforms the baseline methods. - The proposed method is supported by a theoretical analysis presented in Section 4. - The authors have conducted a good number of analyses, including ablation studies and quantification of dynamic mismatch.

Weaknesses

The main weakness of this paper lies in its lack of clarity in communication and presentation. Overall, I found it challenging to follow the arguments and ideas presented. Furthermore, it remains unclear what the significant contributions of this work are and the novel aspects of the proposed filtering mechanism. For instance, in lines 52-58, the authors attempt to summarize their contributions. However, empirically demonstrating the superiority of their method (contribution number 4) is not considered a contribution itself, as every claim requires proper evaluation. Another weakness of this work is the limited breadth of the experimental analysis. While the main results in Figure 4 undoubtedly showcase excellent performance for VGDF in Gym Mujoco environments, it is uncertain whether this generalizes to other environments beyond that scope.

Questions

I strongly recommend that the authors undertake a significant revision of their paper, particularly focusing on Section 4 and Section 5. Additionally, it is important to ensure that the paper is self-contained and does not necessitate reading the appendix. Therefore, I suggest that the authors incorporate additional details regarding the experimental setup from the appendix into the main body of the paper.

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

1: Your assessment is an educated guess. The submission is not in your area or the submission was difficult to understand. Math/other details were not carefully checked.

Soundness

3 good

Presentation

1 poor

Contribution

2 fair

Limitations

The authors seem to have adequately addressed the limitations.

Summary

This paper focuses on the online dynamics adaptation problem where the agent has access to a large number of samples or an offline dataset from a source domain, and must adapt with a smaller number of samples in a target domain. To address this problem, this paper introduces a framework for using a value discrepancy perspective. Specifically, they consider transitions with consistent value targets equivalent, even if they have different dynamics. This is different from prior works that focus purely on dynamics discrepancy, and encourage the agent to avoid state-actions with different dynamics. Towards this end, they show some theoretical analysis demonstrating a bound on policy performance given value consistency. Given this perspective, they present the Value-Guided Data Filtering (VGDF) algorithm which uses value consistency to do selective data sharing from the source domain to the target domain. The practical implementation of this algorithm involves learning an ensemble of gaussian dynamics models for the target domain. These different models are used to generate different fictitious transitions that can be used to estimate a gaussian distribution of potential values in the target domain for a given state-action pair from the source domain. Thus, the consistency of state-actions from the source domain are evaluated by Fictitious Value Proximity (FVP), which computes the likelihood of the estimated value of the source next state given the estimated gaussian distribution of potential values in the target domain. They train a Q function for the target domain by using the samples from the target domain and samples from the source domain if they are in the top $\xi$-quantile likelihood estimation of the sampled minibatch. They train an evaluation policy and exploration policy with SAC, with the main distinction being the exploration policy is trained to optimized the max of the 2 Q functions instead of the minimum. To tackle the setting where the agent only has an offline dataset collected in the source domain, they introduce a BC term to their optimization similar to TD3 + BC. They evaluate their results in several Mujoco environments where the target domain introduces a kinematic shift or morphological shift to the robot. They generally find that their method outperforms their baselines in both the online and offline setting. The ablations demonstrate the benefits of the filtering by FVP and training the separate exploration policy. Finally, they demonstrate how FVP can be used to analyze different types of dynamics shifts in the target domain.

Strengths

The paper is well written, clear, and flows well. I believe an expert could reproduce the algorithm given the details in the paper. I believe the discussion on the value discrepancy perspective is an interesting contribution to the field that would interest other researchers. I think in particular the theoretical results in section 4.2 and the motivating examples and figures in section 4.1 do a great job demonstrating the validity of this perspective. As far as I am aware, VGDF is a novel and interesting algorithm that is well explained and justified by their value discrepancy perspective. The authors include good experimental results and ablations that demonstrate the effectiveness of their algorithm and specific design choices. Additionally, I think the setting of doing online dynamics adaption from an offline dataset is a currently underexplored topic that should get more interest from the research community. The VGDF + BC is a seemingly simple yet effective extension of VGDF that seems to do well in this setting.

Weaknesses

In my opinion, there are no major weaknesses in this paper, but I do have 2 small criticisms I would like to be addressed. I would appreciate comparisons in Section 6.3 to more offline RL algorithms that were designed for offline pretaining and online finetuning, like IQL. I believe there are many real-world scenarios where online samples are prohibitively expensive compared to simulated samples or previously collected offline data. Thus, I would appreciate more ablations with higher numbers for $\Gamma$, but less online samples.

Questions

Do you think your method will be effective in doing sim-to-real transfer? It seems to be a motivating example, but their are no results that directly indicate that your method will be better at sim-to-real transfer, than prior approaches.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

4 excellent

Presentation

3 good

Contribution

3 good

Limitations

No obvious limitations.

Response to the Rebuttal

I appreciate the author's responses. **Regarding (Q1) using TD target instead of next-step value in the implementation of VGDF** Thank you for clarifying the comparison between using Fictitious Value Ensemble (FVE) and Fictitious Value Proximity (FVP) of TD target $r(s, a) + \gamma Q(s', a')$ and using those of next-step value $V(s')$. To prevent any potential confusion among readers, there are a couple of aspects to be clarified in the paper. Firstly, it would be helpful if the authors could elaborate on their reason for choosing to learn the target domain reward function $r(s, a)$ and for providing results based on FVE and FVP of TD target $r(s, a) + \gamma Q(s', a')$ rather than next-state value $V(s')$, even under the assumption of equivalent rewards between domains at the current step. Additionally, it would be beneficial if the authors could address why the differences between the methodology proposed in the paper and its subsequent implementation were not explicitly discussed within the paper itself. The differences seem significant, but the only thing mentioned is that the dynamics model outputs the current-step reward in addition to the next state, as in the appendix. **Regarding (Q2) optimistic exploration $\pi_E$** Thank you for clarifying the approach of using the maximum of the Q ensemble for optimistic estimation. The references in the rebuttal demonstrate the application of the upper confidence bound of Q defined by $\mu_Q + \beta \sigma_Q$, utilizing both empirical mean ($\mu_Q$) and standard deviation ($\sigma_Q$) of the Q ensemble. Employing the maximum of the Q ensemble can be conceptually understood as setting the parameter $\beta$ to 1, a point emphasized in [1]. Given that this approach is already present in the literature, the authors are encouraged to reference the prior works in the paper.

Answer to rebuttal

I thank the authors for their response, and for pointing out the additional experiments in appendix F.5, which I did not find initially, and that are indeed interesting. I had no major concerns initially, and I maintain my original score.

reply

Thank you for the reply, this helps clarify my concerns. I do not plan to update my score but I think the paper is quite reasonable.