Response
Again, thank you so much for responding to our comments
Example:
So, in the example you provided, there is a question about whether the expectation was to reach the goal or to follow the exact path. Let's go with the second case and assume the human's expectation was, in fact, to pass through that exact path. In this case, the robot can never achieve it, as with the reward function. Our method can detect the fact that this cannot be achieved and let humans know that the expectation set cannot be achieved. A purely reward-based system would try to optimize for the best reward estimate (or worse yet, the original reward function); this is what an approach like IRD might do. Even though IRD paper uses the cell with lava as an example, depending on the example, it is possible the average reward need not penalize the lava cells enough to cause the agent to avoid it (for example, there might be other cells of unknown features that weren't there in the human model either). The other advantage of expectations again goes back to the point about transferability and when the expectation set can be satisfied in the robot model. The same reward function might not lead to policies that will satisfy the expectations set in both models. However, different reward functions might exist that would satisfy them in each model. However, the reverse is not true. If the expectation set cannot be achieved in the robot model, no reward function can change that.
Yes, we are assuming that the expectation set captures the true human preferences.
Theorem1 proof:
Again, please note that we are fine with cases where no solutions exist. It is important to identify the problem and inform the user.
So, there are multiple ways to construct a counter-example that shows the absence of a reward function that translates across domains, but policy exists. The simplest one is when you have a case where you take the form of rewards to be $R: S \times A \rightarrow \mathbb{R}$ (note the reward form has nothing to do with the generality of MDP formulation or any of our methods). Now, let us assume the states to be achieved and avoided are ones where no actions are available (again consistent with the most general MDP construction, referred to as control constraint in optimal control [1]). Now, from the starting state, it has access to two actions. One that takes you to the goal state and the other one that takes you to the state to be avoided deterministically. Now, in the human model, you have to reward one over the other. Let's assume in the robot model, their dynamics are reversed. Now, the previous reward function is no longer able to achieve the expectation set in the robot model.
[1] Bertsekas, Dimitri. Dynamic programming and optimal control: Volume I. Vol. 4. Athena scientific, 2012. (http://www.athenasc.com/DP_Slides_2015.pdf page nine for reference)