Apologies for delay, was addressing another deadline.
> R1: We respectfully disagree that our work is more of a reward shaping work, nor can be adapted to the online learning setup, from two perspectives.
> The motivation for decomposing human returns in our case, follows from the analysis in proposition 1 that proper decomposition can help reduce variance and lead to more accurate OPE estimation over policies that lead to varied performance. This is rather different than in existing reward shaping works that try to smooth over sparse rewards, and mainly focus on improving the convergence and/or optimality properties of the optimal policy that can be learned out of the shaped reward functions.
> The focus of our work is to estimate the feedback/returns that are directly provided by human participants, toward the cases where the performance pre-existing RL policies need to be demonstrated before they get deployed to human participants, e.g., as required in healthcare systems. So the problem setup of OPEHF cannot be changed to online. Moreover, the use of bi-directional LSTM requires all trajectories to be fully observed a priori, and cannot be used online as well (see more in R3 below).
I would argue that generally, improving the convergence the policy via reward shaping is *because* reward shaping reduces the variance of the policy update. Perhaps the more accurate way to put it is that both OPEHF and general reward shaping are trying to reduce variance, but one is for evaluation and the other is for improving gradient updates.
I think that overall this is not an important concern though and am willing to move on.
> On RUDDER
The authors are correct that RUDDER's return decomposition view only ensures the optimal policy's return is unchanged. I was more curious at the different variants RUDDER proposes for return decomposition (see Section 3 of paper: integrated gradients, layer-wise relevance propagation, etc.)
Could the authors clarify if OPEHF will maintain the ranks of different policies in the limit of infinite data? It was not clear to me that was a proposed contribution.
> R3: The bi-directional LSTM is used to map the end-of-episode human return back to immediate human rewards (IHRs) over the entire horizon, for all the offline trajectories being considered. Specifically, it is used only after these trajectories are already observed (i.e., no online deployment nor adding new trajectories). Such a case falls under the typical sequence modeling setup where RNNs/LSTMs are mainly used. As an analogy, the use case here is somewhat reminiscent of how decision transformers [3, 4] are used to model the decision process.
I think decision transformers are not a great analogy here. In decision transformers, we condition on the return to go, but this does not change the reward function $r(s,a)$ of the underlying MDP. That reward function is still Markovian / does not depend on the future. Although at training time DT is aware of the future (knows the true return to go), at inference time it is not aware of the future and we pick a return-to-go to condition on at $t=0$.
In OPEHF, the reward $r(s,a)$ does depend on the future, via the bi-directional part of the LSTM. And, this is all fine for off-policy evaluation where we do not do online execution, but I would argue it's not exactly a reward function anymore. You would literally be unable to use the $r(s,a)$ during live inference if I understand correctly.
To be clear: for the purposes of OPE, the bi-directional LSTM is probably helpful. Since the human giving the preference label sees the entire trajectory $\tau$ before rating the trajectory. I just would not describe it as $r(s,a)$ when it is more like $r(s,a|\tau)$ in practice.
> Challenges in getting more than 1 rater for a scenario.
I will quote the reply to Q4 from reviewer ochq.
> Unlike frameworks like RLHF for language model training [5], where one can hire 3rd-party raters to provide feedback simply by following a predefined rubric (without the need of domain expertise); in human-centric domains like healthcare or education, professionals that are thoroughly trained (e.g., doctors, professors, instructors) over years may only be eligible to rate, other than the participants themselves.
I agree with all of this, but think of the question as, *in a hypothetical scenario where there is more than 1 human rater, what would happen?* I am not expecting experiments for this, I understand the setting in this paper has 1 human rater and more would be difficult. I am more looking for discussion on if the method could handle multiple $r(s,a)$ labels gracefully or not. If it cannot it would be a good limitation to mention.
> Evaluation
To be specific, I was interested in comparisons to other methods of return decomposition. i.e. the proposal of this work is, "reconstruct IHR, here is 1 way to do it" and I wanted to see > 1 way of doing it (from the existing data, to avoid needing to get more)