SPRINQL: Sub-optimal Demonstrations driven Offline Imitation Learning

Summary

This paper considers the problem of offline imitation learning from sub-optimal demonstrations. The paper proposes the SPRINQL algorithm, which automatically weights the demonstrations, regularizes the loss with a reference reward function, and then performs inverse soft-Q learning. Experimental results show that the proposed method achieves better performance compared with baselines. Extensive ablations are also provided.

Strengths

The paper is overall well written. The experimental results are strong, and the ablation studies are extensive. It is impressive to see the proposed method improves its performance with more sub-optimal demonstrations provided.

Weaknesses

1. Some parts of the proposed approach are unclear to me. Please see questions 1-4. 2. The influence of the reward regularization term is not completely studied. Please see question 5.

Questions

1. In Equation (3), a modified objective is presented. However, it seems that this objective depends largely on the quality of weights, and it does not necessarily recover the ground truth reward function nor the optimal policy. Given that the weight is chosen with a heuristic, what kind of guarantee can the authors give about the converged policy? 2. Following the first question, in the experiments, seems that the proposed algorithm always outperforms BC-E because of the lack of expert demonstrations. However, I am curious to see what will happen given sufficient expert demonstrations. 3. In equations above line 162, what if $(s, a) = (s', a')$, given that it is possible for two stochastic policies to choose the same action in the same state? 4. Line 173, regarding "prior research", could the authors add some citations? 5. It seems that the weight for reward regularization $\\alpha$ is an important hyperparameter. Will the performance of the proposed method change a lot with different $\\alpha$? In the extreme case, what if we learn a policy with RL using reward function $\\bar r$?

Rating

5

Confidence

4

Soundness

2

Presentation

3

Contribution

3

Limitations

The authors discuss the limitations adequately in the last Section.

Summary

This paper considers the problem of imitation learning on suboptimal demonstrations given coarse trajectory expertise rankings. It introduces SPRINQL, a method combining inverse soft-Q learning, conservative Q learning, and reward learning from ranked trajectory preferences. SPRINQL estimates a reference reward function given trajectory expertise rankings, and extends inverse soft-Q learning to the suboptimal demonstration setting by adding reward regularization and weights based on the estimated reference reward. The paper shows that the SPRINQL objective is tractable with a unique solution over $Q, \pi$. Experiments show that SPRINQL outperforms prior methods on suboptimal trajectories in simulated environments. Ablations justify each component in SPRINQL and demonstrate robustness across a range of dataset sizes and number of suboptimal datasets.

Strengths

- SPRINQL works with a number of suboptimal demonstrations, with reasonable assumptions such as access to trajectory expertise rankings. - The main objective is shown to admit a unique saddle point solution over $Q, \pi$ and is tractable for optimization. - Experiments show that SPRINQL outperforms relevant offline imitation learning baselines on suboptimal trajectories in multiple simulated locomotion and manipulation environments. - Extensive ablations demonstrate that the method works with a range of numbers/sizes of suboptimal datasets, and justify each component in SPRINQL.

Weaknesses

- Suboptimal trajectories are sourced from adding random noise to expert actions. While this is a reasonable proxy, it would be more convincing to see them generated from under-trained RL policies instead. - It would be great to see SPRINQL work on image-based environments.

Questions

- How does SPRINQL perform on suboptimal trajectories generated from under-trained RL policies, instead of rollouts from noisy expert actions?

Rating

7

Confidence

4

Soundness

4

Presentation

4

Contribution

4

Limitations

- SPRINQL assumes access to trajectory expertise rankings.

Summary

This paper presents SPRINQL (Sub-oPtimal demonstrations driven Reward regularized INverse soft Q Learning), a novel algorithm for offline imitation learning with multiple levels of expert demonstrations. SPRINQL leverages distribution matching and reward regularization to effectively leverage the mixture of non-expert data and expert data to enhance learning. An extensive evaluation is conducted to assess SPRINQL, answering eight well-defined questions regarding the capability and performance of SPRINQL.

Strengths

+ The proposed technique presents an improvement for learning from mixed-expert demonstrations and is backed by theory. + The paper is very well-written and contains sufficient detail to understand the key proposed novelties. + The evaluation (including those in the appendix) is extensive and presents strong support for the proposed algorithm.

Weaknesses

- The authors mention that this setting (having several sets of data labeled with relative quality scores) is more general than two quality levels: expert and sub-optimal. Could the authors provide further support for this claim? I would argue that having additional quality annotations makes this setting more niche. - Table 1 contains a very large evaluation but there isn't much commentary on the different trends between baselines and SPRINQL. Comment: The example in the introduction does not seem complete.

Questions

Please address the weaknesses above.

Rating

7

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

Yes

Summary

This work presents an offline imitation learning (IL) method that leverages both expert and suboptimal demonstrations. Distinct from previous methods, this approach assumes access to suboptimal demonstrations across a spectrum of performance levels, with a known ranking to the learners. To utilizes suboptimal demonstations with various levels, this work suggest SPRINQL (Sub-oPtimal demonstrations driven Reward regularized INverse Q-Learning), a reward reguarized inverse soft Q-learning method for the offline IL problem. The evaluation of this method is conducted using the Mujoco and Panda-gym environments, employing a small dataset of expert demonstrations and a larger collection of demonstrations generated by introducing varying levels of noise to expert actions.

Strengths

This work aims to extend the domain of offline imitation learning by addressing the challenge of leveraging imperfect demonstrations with multiple suboptimality levels, presenting a novel and intriguing approach. Drawing on foundational principles from prior research, notably IQ-Learn, this study demonstrates theoretically that the proposed extension, SPRINQL, retains the advantageous properties of IQ-Learn. Furthermore, SPRINQL adeptly incorporates sophisticated concepts such as the Bradley-Terry model and Conservative Q-Learning (CQL) into its training framework. The experimental outcomes are notable, with SPRINQL achieving surprisingly effective results that surpass previous baselines, which were unable to successfully imitate expert performance using the datasets used in this work.

Weaknesses

Significant concerns with this submission are the questionable reporting on the performance of baseline methods and issues related to reproducibility. The experimental results presented are not entirely convincing, as the performance of the baselines substantially deviates from those reported in their original papers. Specifically, even in the case with N=2 suboptimality levels, suitable baselines such as TRAIL, DemoDICE, and DWBC fail to outperform even basic BC-all in nearly all the Mujoco tasks listed in Table 3 (Appendix). The authors need to clarify the reasons behind this anomaly. To enhance the trustfulness of experiments, it would be beneficial for the authors to release the datasets utilized in their experiments or to use standardized datasets such as D4RL or RL-unplugged. This would allow for a more reliable validation of the results and facilitate comparisons with other methods.

Questions

- Could the authors clarify the motivation of employing reference rewards? - Moreover, could you elucidate why reference rewards are utilized as a regularization mechanism rather than directly applying RL using reference rewards? - The problem setting appears closely aligned with preference-based learning, where optimality (or preference) levels are explicitly known. Could you delineate the distinctions between these two problem settings? - Can you provide detailed justifications for why comparing the proposed method with IPL is deemed unsuitable? It appears feasible to compare by segmenting each trajectory by suboptimality levels, then pairing segments from differing levels to facilitate IPL. Could this approach not be applied effectively? [minor comments] reference [15] in line 398 from "NeurIPS 2024" to "NeurIPS 2023."

Rating

5

Confidence

3

Soundness

2

Presentation

3

Contribution

2

Limitations

The authors addressed their limitations in Conclusion section in the main manuscript. They described that this work does not have any direct societal impacts in the Checklist #10, Broader Impacts.