Towards the Transferability of Rewards Recovered via Regularized Inverse Reinforcement Learning

Summary

This paper analyses the problem of the transferability of the reward functions learned from expert demonstrations in regularized MDPs. After having formalized the problem, the authors introduce some assumptions that permit to connect the distance between equivalence classes of reward functions to the suboptimality of the expert. Next, they use the notion of principal angles to relate the distance between transition models with the "amount of information" provided by demonstrations in such settings. Finally, authors propose an algorithm with theoretical guarantees for solving the problem, which they validate empirically.

Strengths

- The paper is written in a clear manner. - The goal is to bring theoretical results (i.e., about known expert's policy) to the real world, by analysing the setting with demonstrations. - The transferability issue is one of the peculiar advantages of IRL over, for instance, BC, thus it is interesting. - The contributions are based on technically solid theoretical results (e.g., authors provide sample complexity analysis of the algorithm as well as convergence analysis).

Weaknesses

- The framework of regularized MDPs makes things much easier w.r.t. the setting of common MDPs. - There are various assumptions along the way (which seem necessary). - Regularized MDPs do not represent a realistic model for expert behavior.

Questions

questions: - Assumption 2.2 is rather strong, because you assume that the transition model is such that, for any arbitrary policy, each state is visited with a fixed non-zero probability. Do you agree? - As you comment in Remark 3.12, it seems clear that all the results depend a lot in how different the considered regularized MDP differs from a common MDP. Since common MDPs are more realistic models to model experts behavior, how limited do you think that your results are when applied to practical applications? minors: - the references should be made aligned with each other, and not like now where some have DOI code, others do not, some have volume, others number of pages, etc... - the checklist shall be put at the end of the appendix typos: - line 130 the operation of difference between equivalence classes is not defined - line 140 it is not clear what is the difference between $N^E$ (which is not defined) and $K$ - line 245 the simplex symbol has $\mathcal{S}$ and $\mathcal{S}\times\mathcal{A}$ reversed.

Rating

7

Confidence

5

Soundness

4

Presentation

4

Contribution

3

Limitations

The authors have adequately addressed the limitations and potential negative societal impact of their work.

Summary

The paper establishes bounds on the reward transferability for inverse reinforcement learning in discrete MDPs with known transition matrix. It is well known that the reward function learned in a single environment is in general not transferable as the reward shaping potential can not be identified and therefore, the recovered reward function may perform poorly under different dynamics. Prior work showed conditions for recovering the true reward function (up to a constant) when observing multiple experts acting in MDPs with different dynamics. Probably most relevant, Rolland et al. (2022) showed that reward identifiability is ensured under some rank conditions on a matrix constructed based on the transition matrixes. However, their analysis assumes that the IRL problem can be accurately solved in the different environments. * The submission extends these previous results by allowing for an error in the recovery of the reward function. Namely, the paper derives a bound based on the principal angles between the subspaces of potential shaping transformations corresponding to the different transformation matrices of the experts. That is, the second largest principle angle between the two observed MDPs, in combination with the optimality gap of the recovered reward function, is used to bound the optimality gap of the optimal policy (wrt the recovered reward function) under any other transition matrix. * Similarly, the paper uses the principle angle to establish a bound on the optimality gap on a given environment, when the reward was recovered in a different environment. * The theoretical results are accompanied with empirical results on a gridworld environment that show the relation between the principle angle and the error of the recovered reward function. The results use projected gradient descent to optimize the reward function based on demonstration under different system dynamics.

Strengths

**Originality** Using prinicpal angles for bounding reward transferability seems to be a novel and useful concept. The previous rank conditions only provided a binary criterium usable for perfect conditions, whereas the principal angles enable us to relate approximation errors to the transferability of the recovered reward. **Significance** The theoretical results could be a step towards better understanding reward transferability also for more realistic settings (e.g. non-finite MDPs, unknown transition matrices). Reward transferability is an important open and critical question in inverse reinforcement learning. Indeed, arguably there is little additional value in a non-transferable reward function compared to an imitation learning policy. **Clarity** Overall, I find the paper well-written and easy to follow. The paper gives a good overview of the work and the theoretical results and assumptions are clear. I also like the geometrical proof Fig 1b a lot. In general the paper avoids unnecessary complications. **Quality** The assumptions (e.g. steepness & convexity of the regularizer and Lipschitz continuity of its gradient, full support for every state-occupancy measure) are reasonable. The claims are well-supported and seem correct (I did not check all derivations in detail).

Weaknesses

**Significance** The paper builds on the concept of principle angles between different subspaces, which can be only computed in simple settings (finite MDPs with known transition functions). While I can imagine that concepts could be transferred to continuous MDPs and to settings where approximations based on transition samples are necessary, it is not fully clear if and how such extension could be accomplished. **Clarity** While overall the paper is clear, I think that the paper should provide some intution about its most relevant concepts. Namely, I am missing an intuition about the subspaces of potential shaping transformations $\text{Im}(E - \gamma T^\top)$, which in turn would be useful for getting an intuition about the principal angles of interest. **Quality** The paper mainly considers the setting when learning with demonstrations from K=2 different transition dynamics. The paper proposes a trivial extension of the bound for K>2, that merely uses the tightest bound among all combinations of two different dynamics. However, I think it can be safely assumed that much better transferability can be guaranteed when learning in K>2 enviroments, as these could complement each other. The bounds are quite loose as evident from Corollary 3.11 in the case of Shannon- and Tsallis-entropy regularization. In particular the term $\exp(H_\gamma)$ should easily explode for the Shannon-Entropy, and similarly the term $H_\gamma^\top$ for the Tsallis-entropy, where the term $H_\gamma$ can easily be in the range of hundreds or thousands for reasonable discount factors. However, I give credit for the fact that the paper mentions this limitation, and at least the large bounds can be somewhat compensated by very small approximation errors (also recovering previous results for $\hat{\epsilon}=0$).

Questions

- Line 133 states that the paper considers equivalence relations induced by constant shifts and the "aforementioned potential shaping transformations". Do these include all general potential shaping transformations? - Could we use the principle angle to design experiments in order to learn transferable reward functions? - Any ideas on how the angles could transfer to continuous MDPs? - How could we estimate them from transition samples? - Could the angles be generalized for K>2?

Rating

8

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

Limitations and potential negative societal impact have been adequately addressed.

Summary

This paper introduces the logical framework regarding rewards identification (up to potential shaping transformations, which can be achieved under entropy regularization) and rewards transferability (up to a constant, more difficult to achieve). The paper considers a practical scenario where only expert demonstrations are accessible, not the full expert policy, and analyzes $\epsilon$-transferability using the principal angles technique. Additionally, when only expert demonstrations are available, they analyze two cases: for two experts, transferring to an arbitrary transition law requires the two laws (from two experts) to be sufficiently different; for a single expert, transferring from the original to the target environment requires the transition laws to be closely related. Finally, they present a probably approximately correct (PAC) algorithm and an end-to-end analysis for learning transferable rewards from multiple expert demonstrations.

Strengths

1. I have particularly appreciated the logical framework regarding rewards identification (up to potential shaping transformations, which can be achieved under entropy regularization) and rewards transferability (up to a constant). I also value the introduction of two methods for attaining rewards transferability: first, by restricting the reward class, such as to state-only rewards, and second, by learning from multiple experts who share the same reward but have different transition laws, provided a specific rank condition is met. 2. The topic selection is meaningful, exemplifying that under the approximate optimality of experts, the learned reward can perform poorly in a new environment, even if the rank condition in Equation (5) is satisfied. 3. They consider a practical scenario where only expert demonstrations are accessible, not the full expert policy, and analyze $\epsilon$-transferability using the principal angles technique. This result encompasses the scenario where the full expert policy is accessible, recovering a reward r^ for which all experts are exactly optimal, with ϵ being 0 for all experts. 4. The comparison between single-expert transfer and multi-expert transfer is insightful.

Weaknesses

1. This work is mainly theoretical. I think experimental validation on real-world applications could provide valuable insight into the practical aspects and challenges of transferability (if possible). 2. I think the example used to describe the rank condition's insufficiency for reward transfer when $\epsilon$ is not exactly 0 (for all experts) is not effective. As $\beta$ tends to 0, the two transition laws become completely different, which is too extreme, despite being easy to follow and aligned with the main theorem. Maybe a toy demo with illustrations is better. 3. When $\epsilon$ is exactly 0 (recover a reward $\hat{r}$ for which all experts are exactly optimal), the work is already established, making the theoretical contribution slightly inferior. I believe a detailed discussion of this work's theoretical innovations and how it differs from the case when the expert policy can be achieved is necessary (from a theory aspect). I will be inclined to increase my score if I appreciate your response.

Questions

1. As the authors mentioned in Section 2, the reward in $\mathcal{U}$ and $\mathcal{V}$ is defined on $\mathcal{S} \times \mathcal{A}$. According to Definition 3.7 (lines 237-239), how are the elements in $\mathcal{U}_{P}, \mathcal{U}_{P'} \in \mathbb{R}^{\mathcal{S} \times \mathcal{A}}$ processed for inner product and modulus operations (to be a constant)? 2. $\|[r]_{\mathcal{V}}-[r']_{\mathcal{V}}\|_{2}$ is small. How to define 'minus' and 'small' needs clarification, as I believe this involves the difference between two sets or two equivalence classes. A detailed explanation would be better. 3. The description of the lower and upper bounds in (6) is unsatisfactory. For the recovered reward, we need to consider both the lower bound (not optimal) and the upper bound (approximately optimal). Therefore, both the lower and upper bounds should be included. 4. I think the implications of Theorem 3.10 should be introduced separately according to single expert and multi experts due to the different meanings of transition law differences. For multiple-experts, the difference in transition laws pertains to the source environments. For single experts, it pertains to the difference between the source environment and the target environment.

Rating

5

Confidence

4

Soundness

3

Presentation

4

Contribution

3

Limitations

The authors have discussed the limitations and future work in Section 6.

Summary

In the context of inverse reinforcement learning (IRL) -- i.e. the task of recovering a reward based on demonstrations from experts acting (approximately) optimally with respect to that reward -- this paper studies the question of transferability to new environment dynamics: if we learn a reward function from demonstrations on environments with dynamics P1 and P2, when can we expect for this reward to lead to the optimal policy on a third environment with dynamics P3? The authors show that previous results guaranteeing transferability do not generalize to cases where we are able to learn the reward only approximately, which would be the case in most practical settings. They then provide two theoretical results giving sufficient conditions for transferability.

Strengths

- The broader question the paper addresses (transferability in IRL) is a key question for the practical usefulness of IRL. IRL papers often claim that the recovered reward is a more generalizable representation of the goal than a policy (that could be recovered e.g. using behavioural cloning, which is usually cheaper). This paper makes a step toward putting such claims on firmer theoretical basis. - The paper clearly shows important limits of prior results in the area. - The authors prove novel theoretical results using an original method. - Once one grasps the necessary formalisms, the rest of the paper has a clear and easy-to-follow structure. - The writing is largely free of typos and other mistakes.

Weaknesses

Only major objection is the lack of pedagogical effort: I think the paper is not optimized to be read by a broader NeurIPS audience. I think it would be easy to read to someone who is (1) an expert on IRL (already a somewhat niche area) and at the same time (2) an expert on the kind of theoretical analysis used in the paper, which leaves only handful of people (I think an audience not big enough to even hold a workshop). Even someone from category (1) with a degree in math and some familiarity with similar theoretical results, I would really appreciate if authors put much more effort into explaining what are the intuitions behind the different abstract concepts that they use. E.g. I think most people active in IRL and RL have much stronger existing intuitions in thinking about policies, rather than occupancy measures, so a slight build-up toward defining stuff in terms of measures would help them. Similarly when introducing a "strictly convex regularizer" it may be useful to give an intuition what is the purpose of that regularizer .Often adding a single sentence in places like this would make a huge difference to many readers. Also defining the transition operator as $(Pv)(s,a) = \sum_{s'} P(s'|s,a)v(s')$ is about as unintuitive as it gets for an otherwise so intuitive concept.

Questions

I would appreciate more discussion on how you would expect similar results to generalize to continuous state and action spaces - you title the paper "Towards the Transferability ..." so it would be good to have more discussion of whether and how this simple finite linear case is a useful step toward establishing more practically useful results. This is key for assessing the importance of this paper.

Rating

6

Confidence

4

Soundness

4

Presentation

2

Contribution

3

Limitations

The results in the paper are limited to only finite state and action spaces and furthermore to recovering reward from an expert whose policy is corresponding to a particular kind of regularization. It is unclear whether this kind of regularization anyhow corresponds e.g. to how a human would demonstrate a practical task to a robot (I would say it doesn't).

Follow-up

Apologies, we realized that we missed addressing the following question: **(C)** "_Assumption 2.2 is rather strong because you assume that the transition model is such that, for any arbitrary policy, each state is visited with a fixed non-zero probability. Do you agree?_" **(A)** Yes, that's correct. The assumption could be ensured, for instance, by a lower bound on the initial state distribution. Moreover, in our analysis, we only require it to hold for the optimal policies under the experts' and the recovered reward. However, it would be interesting to explore whether this assumption could be replaced with weaker notions of coverability or concentrability.