Sequential Decision Making with Expert Demonstrations under Unobserved Heterogeneity

Summary

Imitation learning in a setting where experts have contextual information not available to the imitator. Imitator sees state-action (no reward) trajectories for the expert data. The paper proposes to use the expert data as prior and then do meta-RL to learn a policy.

Strengths

- The problem setting considered is important and there isn't too much prior work in it. - The proposed method seems sensible. - The paper provides theoretical and empirical justification for the proposed method.

Weaknesses

- The empirical evaluation is limited to simple domains and it is not discussed why that is the case. It would be good to include an explanation on why this was not scaled up to some more complex environments. - Comparison to ExPLORe: if there is a way to use the same actor-critic algorithm for the proposed method and the comparison that would strengthen the paper. As the text says, now it is not clear whether the difference comes from the online RL part or from the difference in how they use the expert data. - ADVISOR [1] targets the same problem setting. Differences to that algorithm should be discussed. And maybe a pointer should be given on why their method is demonstrated in high-dimensional domains and this isn't. Not saying that this has to be evaluated in high-dimensional domains, but that it is not immediately clear why this isn't. - Figure 4 - Explain the pane meanings explicitly in the caption. - Explain the axes explicitly in the caption. I.e., are these learning curves? - How come the naive baseline does worse in the low entropy setting than in the high entropy one? [1] Luca Weihs, Unnat Jain, Iou-Jen Liu, Jordi Salvador, Svetlana Lazebnik, Aniruddha Kembhavi, and Alex Schwing. Bridging the imitation gap by adaptive insubordination. Advances in Neural Information Processing Systems, 34:19134–19146, 2021.

Questions

- Why is human-in-the-loop highlighted as a limitation? - Can you give a more fleshed out example of how the problem of unobserved contextual information might appear in a realistic problem? - Can you lay out an idea on how to scale the method?

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

Limitations are discussed but some of the limitations highlighted seem a little strange. See questions.

Summary

The paper proposes a 2-stage learning method for sequential decision making, wherein step 1 involves leaning a prior from expert demonstrations and step 2 is online RL where that learnt prior is used. The main problem being addressed is that of heterogeneity of data and/or contexts provided by the expert and those encountered by the online learning agent.

Strengths

- Relevant problem: Finding useful, scalable and efficient ways to leverage offline expert demonstrations for online decision making is of key interest to the community. - The proposed method combines imitation learning with online model-based RL, paying attention to the issue of heterogeneity between the available contexts that gave rise to the offline data, that might be unavailable during the online RL step. - The problem is well-motivated and the approach is easy to understand conceptually; contributions are clearly outlined. - Presentation (writing, figures etc) is mostly clear, with some minor comments included in the below section.

Weaknesses

Clarity - Notation: even with the notation paragraph, I find it quite difficult to follow. E.g. I think some `t` superscripts are missing (the definition of the q-function); should $\mathcal{C}$ be part of the definition of the MDP? - Terminology: I'm a little confused with the terminology "parametric" vs "non-parametric". Unless I'm missing something, the latter is still parametric as you are optimising over a family of distributions? - Typo: in figure 1, annotation below "Step 3" is repeated, I guess one should be $\mu_{\theta}*$. - Figure 1: clarity can be improved if you write down what is being learnt and what is being updated. - I found the algorithm box quite useful, will be great to have it in the main paper (e.g. next to Figure 1). Max-entropy prior: - Clearly, if $\mu_0 \in \mathcal{P}(\epsilon)$, then we get a trivial solution, $\mu_{ME}=\mu_0$ (usually Uniform), so do you need an assumption on ${P}(\epsilon)$ to get a non-trivial solution? - If $\mathcal{C}$ is unbounded, then you can't really define a uniform distribution on it, how do you deal with that? - Can you handle improper priors?

Questions

In addition to those raised in the Limitations section: 1. Do you have an intuition on how badly "misspecified" the learnt expert prior can get until learning for the online agent becomes very difficult? Is there a way to detect a poorly specified prior, and potentially reverting to an alternative (e.g. uninformative) after a few steps online? 2. For the "parametric" ExPrior - why score gradients? Have you thought about or tried continuous relaxation when the problem is discrete? 3. On line 112 "Note that since the task variable is unobservable, the learner's policy will not depend on it" - does it depend implicitly though, e.g. can the learner infer and integrate over that inferred task distribution?

Rating

5

Confidence

2

Soundness

3

Presentation

2

Contribution

2

Limitations

There are no specific negative societal impact that need to be discussed.

Summary

This paper attempt to leverage offline demonstrations to speed up online learning under unobserved heterogeneity, and unlike zero-shot meta reinforcement learning, the proposed ExPerior does not require the task labels (reward labels). ExPerior utilizes expert data to establish an informative prior distribution for online exploration. The experimental results on multi-armed bandits and MDPs showcase the superiority against current baselines.

Strengths

The setting proposed in this paper is realistic and significant. To solve the problem that the learner faces uncertainty and variability in task parameters, the paper proposes two approaches, parametric and non-parametric, and provides sufficient theoretical basis. Experiments under multi-armed bandits and MDPs also prove its effectiveness.

Weaknesses

1. Just as stated at the end of the paper, the experiments conducted in the paper are limited, and the experiments under MDPs even made the assumption that the transition functions are invariant to the task variables, which makes the experimental environment overly simple. In more difficult experimental benchmarks such as MuJoCo, it is questionable whether the proposed method can show the expected effect. 2. Due to the deficiency mentioned in the first point just now, the paper also cannot further compare with modern offline meta RL algorithms (such as PEARL, FOCAL, CORRO) under complex MDPs, making the baselines seem a bit scarce. 3. The introduction of related concepts in the paper is slightly insufficient. Besides, there is a clerical error in the upper right part of Fig. 1: the first \mu_{\rm{ME}} should be \mu_{\theta^*}.

Questions

1. What does "Heterogeneity" mentioned in the paper represent? Does it refer to the differences between the transition functions and reward functions in MDPs? The paper should explain this. 2. The paper mentions that the off-policy meta RL method PEARL requires task labels. Does the task label here refer to the reward label? In fact, the trajectories required by PEARL only contain (s, a, r, s') information.

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

A/N

Summary

This paper studies online transfer learning in an unknown Markov decision process. The learner has access to demonstration trajectories generated by an expert, who has access to an independent context variable. The expert can observe the values of the latent context to make a decision, but such information is unobserved to the learner. The authors propose a learning strategy that allows the learner to accelerate its online learning process by leveraging the confounded expert's trajectories. More specifically, the expert selects values of the action following a specific parametric family of sofmax policy. Using this parametric information, the learner can extrapolate an informative prior about the underlying system dynamics (i.e., transition function and reward function) from the confounded trajectories and utilize the extrapolated prior to improve future learning. The authors then derive a Bayesian regret bound for the proposed method. Simulation results support the proposed approach.

Strengths

- The paper is well-organized and clearly written. Simulations are comprehensive, supporting the proposed approach. - Theoretica regret analysis is provided. I have not read through all the proofs for Theorem 2. But the result seems reasonable.

Weaknesses

- The proposed method requires the expert to follow a specific form of policy to generate the demonstration data. This parametric restriction is rather strong and may not necessarily hold in many applications. - It is unclear to see from Theorem 2 under which condition the expert's demonstration could accelerate the online learning process. It would be appreciated if the authors could further elaborate on this point. - Some related references are missing. Extrapolating knowledge from confounded data is one of the main problems in causal inference. There is a growing line of work studying learning from confounded demonstration data in canonical RL tasks, including: 1. Kallus, Nathan, and Angela Zhou. "Confounding-robust policy improvement." Advances in neural information processing systems 31 (2018). 2. Zhang, Junzhe, and Elias Bareinboim. "Near-optimal reinforcement learning in dynamic treatment regimes." Advances in Neural Information Processing Systems 32 (2019).

Questions

- Could the regret bound in Theorem 2 outperform standard bandit regret when the expert's demonstration is benign? Could the authors elaborate on this?

Rating

5

Confidence

3

Soundness

3

Presentation

3

Contribution

2

Limitations

The authors have adequately addressed the limitations of the paper.