Response
We thank reviewer uyUs for their feedback! We have responded to your questions and concerns below:
Weaknesses:
1. The intended use for these metrics is to *evaluate* reward learning algorithms, in a setting where the true reward function *is* known. For example, we may randomly generate reward functions, use a reward learning algorithm, and then measure how different the learnt reward function is from the true reward function. Alternatively, they can also be used in theoretical work, where we may wish to bound the distance between the true reward and the learnt reward (given different amounts of training data, for example). Such analysis would then indicate how reliable a given reward learning method is, which in turn then tells us how accurate we should expect a learnt reward function to be, in a setting where the true reward function is not known. Note that Gleave et al is cited 37 times, and Wulfe et al is cited 6 times; these references will provide examples of practical situations where reward function pseudo-metrics can be used. For example, see eg Adeniji et al, 2023, Figure 1 in Sanghvi et al, 2021, or Figure 4(c) in Liang et al, 2022.
2. We do not require a simulator with the true transitions --- samples from the true transitions are sufficient. For example, the VAL canonicalisation can be estimated using samples. This is discussed in the first paragraph of Section 2.3.
3. We agree that this may be a limitation in some contexts. However, note that the error can be reduced by increasing the number of samples used in the estimation. Moreover, it can be eliminated completely if the STARC metric is computed exactly (although this may not be feasible in large environments).
4. We agree that the spread between L and U is important. However, note that any metric which is sound and complete must be bilipschitz equivalent to STARC metrics -- this means that whichever metric has the smallest spread between L and U must be quite similar to STARC metric (and it might be an instance of a STARC metric). Note also that the exact value of L and U depends on the transition function; to see this, consider the second figure in Appendix B (the transition function affects the diameter of $\Omega$).
Questions:
1. Yes, this is indeed a typo, thank you for pointing this out.
2. We describe EPIC in Section 1.2. We can make this description more extensive, to make it easier to understand.
3. In our experiments, $D_S$ and $D_A$ were both uniform for EPIC. We will make sure that this is included in Appendix F; thank you for spotting this omission.
4. $S'$-redistribution is simply any transformation that never changes the expected value of the reward function. For example, suppose we have a nondeterministic MDP where an action $a$ in state $s$ leads to $s_1$ or $s_2$ with equal probability, and that $R_1(s,a,s_1) = 1$, $R_1(s,a,s_1) = 0$. Then $S'$-redistribution could be used to produce a reward function $R_2$ where $R_2(s,a,s_1) = R_2(s,a,s_2) = 0.5$.
5. The conditions in Definition 1 are meant to hold for all reward functions -- it can thus be read as "for all $R$, ..., and for all $R_1, R_2$, ... We will amend the text to make this more clear.
6. $\mathrm{Im}(c)$ is the output of the canonicalisation function, when it is applied to the space of all reward functions. From the definition of canonicalisation functions, we get that this is a linear subspace of $\mathcal{R}$ in which no reward functions differ by potential shaping or $S'$-redistribution. A geometric description of this is given in Appendix B, but we can also describe this in more detail in the main text.
7. Yes. We will amend the text to make this clear.
8. For example, relative to the $L_1$-norm or the $L_\infty$-norm, there will be multiple minimal canonicalisation functions. This is because the unit balls of these norms are not strictly convex.
9. We expect that it in practice often will be best to pick $n$ and $m$ so that they can be computed easily. Our experiments suggest that the $L_1$-norm is the best (unweighted) norm for both $n$ and $m$.
10. VAL is a valid canonicalisation function for any choice of policy, so we simply used a policy for which $V^\pi$ would be easy to learn. The benefit of using a uniform policy, rather than some deterministic policy, is that exploration then is built in automatically.
We hope that this clarifies our paper, and that the reviewer might consider increasing their score.