Hindsight Preference Learning for Offline Preference-based Reinforcement Learning

Thank you for your valuable evaluations and suggestions. Below we provide further explanations for the concerns and questions about our paper. - **W1: The distribution shift between $\mathcal{D}\_{\rm u}$ and $\mathcal{D}\_{\rm p}$ may lack generality.** We would like to note that, although in some cases $\mathcal{D}\_{\rm p}$ is sampled from $\mathcal{D}\_{\rm u}$, their magnitudes are very different since crowdsourced annotation can be expensive. In our experiments, even if the $\mathcal{D}\_{\rm p}$ is sampled from $\mathcal{D}\_{\rm u}$, the amount of data in $\mathcal{D}\_{\rm p}$ is <10% of $\mathcal{D}\_{\rm u}$, making $\mathcal{D}\_{\rm p}$ a possibly biased subset of $\mathcal{D}\_{\rm u}$. Besides, in PbRL the queries in $\mathcal{D}\_{\rm p}$ are often actively selected based on some heuristics to maximally gather information with less budget (see [1] as an example). Such non-uniform sampling inherently causes a mismatch between the datasets. In addition, the preference distribution shift has also been identified in other works. For example, [2] discovers that the misalignment between queries and the current policy can hinder learning efficiency, and they propose to sample preference data (for human labeling) from the most recent experiences collected by the policy. At a larger and more practical scale, in LLM RLHF fine-tuning, it was found that the reward model can quickly degrade if not exposed to the sample distribution of updated language models, thus necessitating an iterative data collection procedure [3]. These findings collectively demonstrate the effect of preference shifts. [1] Daniel Shin, et al. "Benchmarks and Algorithms for Offline Preference-Based Reward Learning." [2] Xiao Hu, et al. "Query-policy misalignment in preference-based reinforcement learning." [3] Hugo Touvron, et al. "Llama 2: Open Foundation and Fine-Tuned Chat Models" - **W2: Theoretical analysis.** Thanks for your insight and recommendation! We will consider and look into the theoretical characterization of HPL. - **W3: Performance regarding OPPO and CPL.** Yes, we believe the limited size of the preference dataset is the primary reason for the unsatisfactory performance of CPL, despite our effort to augment CPL with BC regularizations. Please refer to line 1056 for details. For OPPO, we employed the open-sourced code by the authors without any alterations and utilized the same hyperparameters from the original paper. However, we observed significant fluctuations in the learning process and declining performances over time. - **W4: Figure 6c, increasing the size of $\mathcal{D}_{p}$ leads to worse performance, why is that?** Figure 6 is about the analysis with the VAE, so we think you are actually referring the Figure 7c where reducing $\mathcal{D}\_{\rm p}$ instead leads to better performance for both MR and HPL. The trajectories in the preference dataset are collected by three types of policy checkpoints: expert-level policy, expert policy with 1.0-std Gaussian perturbation, and random policy. Consequently, the quality and value of each trajectory can vary significantly. For $|\mathcal{D}\_{\rm p}|=100$, we conjecture that the proportion of those bad trajectories is comparatively higher than $|\mathcal{D}\_{\rm p}|=50$, which hurts the overall quality of the reward function. A similar phenomenon was also observed in the paper that provides the dataset (see Table 3 in [4]). [4] Joey Hejna, et al. "Inverse Preference Learning: Preference-based RL without a Reward Function." - **W5.1: using $\mathcal{D}_{\rm p}$ to learn representations as a baseline** Thanks for pointing this out. To verify this, we implemented another variant of HPL where the representation networks and the prior distribution are learned using $\mathcal{D}\_{\rm p}$. We trained for 3 seed and the performances are listed in the following table. The results for the variant that uses $\mathcal{D}\_{\rm u}$ for marginalization are from the draft. ||hop: med-e → med-r|hop: med-r → med-e|walk: med-e → med-r|walk: med-r → med-e| |:---:|:---:|:---:|:---:|:---:| |representation from $\mathcal{D}_{\rm p}$|43.0±27.5|75.6±7.6|66.1±9.2|108.2±0.2| |representation from $\mathcal{D}_{\rm u}$|71.1±8.3|99.5±13.8|71.8±7.7|108.4±0.6| - **W5.2: Bi-directional PT as another baseline.** The Preference Transformer consists of a stack of causal attention layers, followed by one bidirectional attention layer at the end to calculate the importance weights of each state-action pair (see Figure 2 left in [5]). Therefore, the PT used in our experiment already incorporates future information. [5] Changyeon Kim, et al. Preference Transformer: Modeling Human Preferences using Transformers for RL.

Thank you for spending time reviewing our paper and also for your appreciation. Below we provide further explanations and hope they can address your concerns. - **W2: Fail to address potential shifts in visual appearance, dynamics and sensor noise.** We acknowledge that practical applications of reinforcement learning agents must address additional challenges, such as distribution shifts in visual observations, and the community has made significant efforts to tackle these issues. However, this paper primarily focuses on the specific distribution shift between the preference dataset and the unlabeled dataset. We believe that the proposed method can be combined with other techniques to collectively enhance the robustness of RL systems. - **Q1: Preference distribution shift on real-world, large-scale datasets?** The preference distribution shift problem has also been identified in other studies. For example, [1] discovers that the misalignment between queries and the current policy can hinder learning efficiency, and they propose to sample preference data (for human labeling) from the most recent experiences collected by the policy. At a larger and more practical scale, in LLM RLHF fine-tuning, it was found that the reward model can quickly degrade if not exposed to the sample distribution of updated language models, thus necessitating an iterative data collection procedure [2]. These findings collectively demonstrate the effect of preference shifts on real-world and large-scale scenarios. [1] Xiao Hu, et al. "Query-policy misalignment in preference-based reinforcement learning." [2] Hugo Touvron, et al. "Llama 2: Open Foundation and Fine-Tuned Chat Models." - **Q2: Effect of $N$** We carried out experiments using the MuJoCo-Gym tasks to ablate the effect of $N$, and found that generally HPL is not sensitive to this value: ||hop-med-rep|hop-med-exp|walk-med-rep|walk-med-exp| |:---:|:---:|:---:|:---:|:---:| |N=1|79.91±9.29|107.99±4.40|60.40±3.78|105.78±3.28| |N=5|86.62±4.80|100.61±0.98|64.71±4.59|104.26±3.00| |N=20|78.89±7.48|103.78±6.65|60.82±3.46|107.93±1.77| |N=50|81.74±8.08|105.05±3.03|64.42±6.87|108.93±0.51| It is also worth noting that the sampling cost is amortized and negligible, as we only need to label the rewards once before the reinforcement learning (RL) training begins. - **Q3: Similarity to retrieval-based offline RL methods.** Thanks for pointing this out. The idea of leveraging trajectory segments and learning compact representations for them indeed shares similarities in methodologies with retrieval-based offline RL methods. The recommended papers both build a graph where the original states in the MDP are categorized into vertices in the graph based on learned metrics or representations, and the vertices are connected via the actions. With such a graph, [1] performs value iteration on this graph-based MDP, while [2] retrieves neighbors and trains the imitation policy using weighted behavior cloning. Nevertheless, HPL focuses on leveraging future segments for improved reward learning, rather than retrieving memory for imitation. We will add one section to discuss the similarities and connections between HPL and other demonstration-involved RL methods (e.g. retrieval-based RL, imitation RL) in the revised version.

Thank you for your time in reviewing and catching the typo. Below we provide further explanations and experiment results for your reference. - **W1: It remains unclear whether hindsight information was effectively applied to achieve accurate credit assignment.** Following existing practices in PbRL, we carried out Pearson analysis with the reward model obtained by MR and HPL. To be specific, we used datasets from Gym-MuJoCo tasks, and labeled the preference according to the trajectory return and used HPL/MR to learn the reward model. HPL additionally uses the vast unlabeled dataset to learn representations and the prior distribution. Afterward, we used the learned reward models to predict the rewards of data points in the unlabeled dataset and calculate the Pearson correlation coefficient between the predictions and the ground-truth rewards. The results are listed in the Table: ||HPL Pearson|MR Pearson|HPL Performance|MR Performance| |---:|:---:|:---:|:---:|:---:| |hop-med-rep|0.85|0.81|80.17±13.61|48.27±14.53| |hop-med-exp|0.55|0.49|103.79±6.03|83.93±37.08| |walk-med-rep|0.81|0.81|82.86±1.40|73.66±4.52| |walk-med-exp|0.52|0.46|110.72±0.55|102.53±13.59| We observed that the rewards provided by HPL are more consistent with the ground-truth rewards, and this enhanced consistency translates into improved performance. These findings further highlight the effectiveness of HPL. - **W2: About the training speed.** Yes, HPL is slower than other methods. We would like to note that the training time shown in Figure 10 includes the training of VAE, which takes up most of the time for reward model training. In comparison, the time spent on summing all steps of each trajectory and sampling is negligible. - **W3: How can the limitation about length $k$ be addressed** The VAE is used to generate a compact representation for the future segments to facilitate sampling and training. This motivation also echoes some other literature, e.g. Hindsight Information Matching (HIM) [1], which employs a Decision Transformer-like pipeline to obtain the representations and demonstrates good performance in both offline RL and PbRL tasks. Therefore, techniques like HIM may be used to learn representations of longer sequences. Additionally, domain-specific knowledge can inspire more effective ways to represent future segments. For instance, in navigation tasks, the intended goal state could be used as a condition, offering a more tailored and effective variant of HPL for such scenarios. [1] Hiroki Furuta, et al. Generalized Decision Transformer for Offline Hindsight Information Matching. - **Q1: Effect of $N$.** We carried out experiments using the MuJoCo-Gym tasks to ablate the effect of $N$, and found that generally HPL is not sensitive to this value: ||hop-med-rep|hop-med-exp|walk-med-rep|walk-med-exp| |---:|:---:|:---:|:---:|:---:| |N=1|79.91±9.29|107.99±4.40|60.40±3.78|105.78±3.28| |N=5|86.62±4.80|100.61±0.98|64.71±4.59|104.26±3.00| |N=20|78.89±7.48|103.78±6.65|60.82±3.46|107.93±1.77| |N=50|81.74±8.08|105.05±3.03|64.42±6.87|108.93±0.51| It is also worth noting that the sampling cost is amortized and negligible, as we only need to label the rewards once before the reinforcement learning (RL) training begins. - **Q2: Using both preference data and unlabeled data to train VAE?** For most of our experiments (except for the mismatch ones), the preference dataset is sampled from the offline dataset (1% ~ 10%), and therefore there is no need to mix them. Besides, we hope the data distribution to resemble the one used for RL training, and thus we decided to use the unlabeled dataset.

Thank you for your time in reviewing our paper. Below we provide explanations about certain concerns or questions you raised in the review, and we are willing to engage in further discussions if you have further questions. - **W1: Q functions, rather than the reward functions, should be dependent on future information. Besides, Downstream RL algorithms utilize $r_\psi$ as the reward function, employing a method that does not consider future information.** Theoretically, yes, ideal rewards (or ground-truth rewards) should depend solely on the current state-action pair and remain independent of the data distribution. However, since the reward model is learned using preference labels derived from entire trajectories, the learning process inherently introduces a dependence on the training data distribution, as demonstrated in the gambling MDP example. What HPL is doing is to correct the dependence on training data, by 1) making the dependence on the future explicit as conditions and 2) marginalizing the conditions using the target distribution (equation 9 in the paper). In this way, the predicted rewards are more aligned with $\mathcal{D}_{\rm u}$, leading to improved performance during downstream RL optimization. HPL does not aim to leverage future information in the RL stage, as the rewards are already corrected and aligned with the unlabeled dataset. - **Q1: The preference signals on the MetaWorld platform do not consider future information.** Similar to W1, the preference signals are assigned by evaluating the goodness of the whole trajectory and this causes the learned reward to have an implicit dependence on the rest part of the trajectories.

Thank you for your valuable evaluations and suggestions. - **W1: Positioning HPL within the research on non-Markovian rewards.** HPL still assumes the Markovian property, i.e. the ground truth reward only depends on the current state and action, rather than the history. Hindsight PRIOR also makes such an assumption, as it learns a reward model that takes the state and action as input (we mistakenly stated that Hindsight PRIOR uses non-Markovian rewards in the draft), while Preference Transformer employs non-Markovian rewards predicted by transformer layers. We think the primary challenge that HPL hopes to address is how to attribute rewards to individual state-action pairs when preferences are based on evaluating an entire trajectory. HPL does this by first assigning credits to conditional rewards and then marginalizing the conditions. Preference Transformer and Hindsight PRIOR, on the other hand, adjust the importance weights of each state-action pair using some attention scores. It is also applicable to extend HPL and learn non-Markovian rewards, by using RNNs to encode the history into an embedding $h_t$ and feeding this embedding to the VAE networks. The ELBO objective (Eq. 7) remains unchanged, similar to how previous works (e.g. PlaNet, Dreamer) train the RSSM models. - **W2: Comparison to Hindsight PRIOR** Hindsight PRIOR hypothesizes that the attention scores between the last state (as queries) and the current state-action pair (as keys) can measure the importance weight and redistribute the rewards. This heuristic approach may be challenged in situations where the assumption may not hold, for example, when attention to the last state is irrelevant to the preference. HPL does not incorporate such heuristics into preference modeling. Instead, it considers a scenario where a trajectory is preferred not because the current state and action are good, but because the agent made rewarding decisions in subsequent steps. Future-conditional rewards can capture and separate the dependence on the future, and reflect the actual utility of the state-action pair. Similar motivations are also shared by literature on credit assignment, e.g. [1]. [1] Thomas Mesnard, et al. "Counterfactual Credit Assignment in Model-Free Reinforcement Learning." - **W3: Modeling difficulty in more stochastic environments.** In more stochastic environments, the distribution of future segments can be highly complex due to the stochasticity of both the policy and the dynamics. We agree that the current approach (using VAE to compress future segments), although general, can face challenges in summarizing the future as $k$ grows. However, we note that it is possible to design more effective ways to represent future segments with domain knowledge. For instance, in navigation tasks, the intended goal state could be used as a condition, providing a more targeted and meaningful summary. - **Q1: Why do the encoder, the decoder and the prior share the notation of parameters?** We use $\theta$ to denote the collections of the parameters of the encoder, decoder, and the prior. They are also jointly optimized using the ELBO loss. - **Q2: How does that distribution mismatch between the datasets cause problems? How does HPL address this issue?** Humans often assess the desirability of a sequence of actions by considering the overall outcome, and this gives the learned reward model an implicit dependence on the training data distribution ($\mathcal{D}\_{\rm p}$). HPL mitigates this issue by making the dependence explicit as the condition and marginalizing the condition using statistics from the target distribution ($\mathcal{D}_{\rm u}$). To demonstrate the improved credit assignment of HPL quantitatively, we followed existing practices in PbRL and carried out Pearson analysis with the reward model obtained by MR and HPL. Specifically, we used datasets from Gym-MuJoCo tasks, **labeled the preference according to the trajectory return**, and used HPL/MR to learn the reward model. HPL additionally uses the vast unlabeled dataset to learn representations and the prior distribution. We then used the learned reward models to predict the rewards for data points in the unlabeled dataset and calculated the Pearson correlation coefficient between these predictions and the ground-truth rewards. The results are listed in the Table: ||HPL Pearson|MR Pearson|HPL Performance|MR Performance| |:---:|:---:|:---:|:---:|:---:| |hop-med-rep|0.85|0.81|80.17±13.61|48.27±14.53| |hop-med-exp|0.55|0.49|103.79±6.03|83.93±37.08| |walk-med-rep|0.81|0.81|82.86±1.40|73.66±4.52| |walk-med-exp|0.52|0.46|110.72±0.55|102.53±13.59| We observed that the rewards provided by HPL are more consistent with the ground-truth rewards, and this enhanced consistency translates into improved performance. These findings further highlight the effectiveness of HPL.