Library

EvIL: Evolution Strategies for Generalisable Imitation Learning

Often times in imitation learning (IL), the environment we collect expert demonstrations in and the environment we want to deploy our learned policy in aren't exactly the same (e.g. demonstrations collected in simulation but deployment in the real world). Compared to policy-centric approaches to IL like behavioural cloning, reward-centric approaches like inverse reinforcement learning (IRL) often better replicate expert behaviour in new environments. This transfer is usually performed by optimising the recovered reward under the dynamics of the target environment. However, (a) we find that modern deep IL algorithms frequently recover rewards which induce policies far weaker than the expert, even in the same environment the demonstrations were collected in. Furthermore, (b) these rewards are often quite poorly shaped, necessitating extensive environment interaction to optimise effectively. We provide simple and scalable fixes to both of these concerns. For (a), we find that reward model ensembles combined with a slightly different training objective significantly improves re-training and transfer performance. For (b), we propose a novel evolution-strategies based method EvIL to optimise for a reward-shaping term that speeds up re-training in the target environment, closing a gap left open by the classical theory of IRL. On a suite of continuous control tasks, we are able to re-train policies in target (and source) environments more interaction-efficiently than prior work.

cs.NEcs.LGComputer Science
Source

Paper

References (60)

01Behaviour DistillationAndrei Lupu, Chris Lu, Jarek Liesen et al.2024 · International Conference on Learning Representations · 8 citations In Library
02Hybrid Inverse Reinforcement LearningJuntao Ren, Gokul Swamy, Steven Wu et al.2024 · International Conference on Machine Learning · 36 citations In Library
03Discovering Temporally-Aware Reinforcement Learning AlgorithmsMatthew Jackson, Chris Lu, Louis Kirsch et al.2024 · International Conference on Learning Representations · 25 citations In Library
04Scaling Opponent Shaping to High Dimensional GamesAkbir Khan, Timon Willi, Newton Kwan et al.2023 · Adaptive Agents and Multi-Agent Systems · 15 citations In Library
05Toward Computationally Efficient Inverse Reinforcement Learning via Reward ShapingL. H. Cooke, Harvey Klyne, Edwin Zhang et al.2023 · Tiny Papers @ ICLR · 2 citations In Library
06Waymax: An Accelerated, Data-Driven Simulator for Large-Scale Autonomous Driving ResearchCole Gulino, Justin Fu, Wenjie Luo et al.2023 · Neural Information Processing Systems · 218 citations In Library
07Massively Scalable Inverse Reinforcement Learning in Google MapsMatt Barnes, Matthew Abueg, O. Lange et al.2023 · International Conference on Learning Representations · 12 citations In Library
08Bridging RL Theory and Practice with the Effective HorizonCassidy Laidlaw, Stuart J. Russell, A. Dragan2023 · Neural Information Processing Systems · 41 citations In Library
09evosax: JAX-Based Evolution StrategiesR. Lange2022 · GECCO Companion · 85 citations In Library
10Adversarial Cheap TalkChris Xiaoxuan Lu, Timon Willi, Alistair Letcher et al.2022 · International Conference on Machine Learning · 17 citations In Library
11Discovered Policy OptimisationChris Lu, J. Kuba, Alistair Letcher et al.2022 · Neural Information Processing Systems · 131 citations In Library
12Nocturne: a scalable driving benchmark for bringing multi-agent learning one step closer to the real worldE. Vinitsky, Nathan Lichtl'e, Xiaomeng Yang et al.2022 · Neural Information Processing Systems · 76 citations In Library

Scroll for more · 38 remaining

Similar papers

RILe: Reinforced Imitation LearningMert Albaba, S. Christen, Christoph Gebhardt et al.2024 · arXiv.org · 2 citations In Library
Energy-Based Imitation LearningMinghuan Liu, Tairan He, Minkai Xu et al.2021 In Library
Energy-Based Imitation LearningMinghuan Liu, Tairan He, Minkai Xu et al.2020 · arXiv (Cornell University) · 55 citations In Library
Sample Efficient Imitation Learning via Reward Function Trained in AdvanceLihua Zhang2021 · arXiv (Cornell University) · 1 citation In Library
Robust Learning from Observation with Model MisspecificationLuca Viano, Yu‐Ting Huang, Parameswaran Kamalaruban et al.2022 In Library
Robust Learning from Observation with Model MisspecificationLuca Viano, Yuting Huang, Parameswaran Kamalaruban et al.2022 · arXiv (Cornell University) In Library
Bridging the Gap Between Imitation Learning and Inverse Reinforcement LearningBilal Piot, M. Geist, O. Pietquin2017 · IEEE Transactions on Neural Networks and Learning Systems · 95 citations In Library
Planning for Sample Efficient Imitation LearningZhao-Heng Yin, Weirui Ye, Qifeng Chen et al.2022 · arXiv (Cornell University) · 32 citations In Library
Prioritizing Expert-Like Transitions for Reward-Free Offline Imitation LearningZhiliang Lin, Zhuangzhuang Chen, Guanming Zhu et al.2026 · IEEE Transactions on Automation Science and Engineering · 0 citations In Library
Coherent Soft Imitation LearningJoe Watson, Sandy H. Huang, Nicolas Heess2023 · arXiv (Cornell University) · 25 citations In Library

Peer review

RejectICLR 2024 · via OpenReview
Reviewer foS13/10 · confidence 5/52023-10-26

Summary

This paper presents a method of model-based imitation learning based on learning and evolution. Specifically, the proposed method named EvIL is formulated as a bilevel optimization problem. The outer loop optimizes the reward and transition functions that are used to train a policy by reinforcement learning in the inner loop. The objective function of the outer loop is behavioral cloning loss, and CMA-ES is adopted to optimize the reward and transition functions. The evolved reward function is distilled to simplify the network architecture of the reward function, which improves the performance in the test environment. EvIL is evaluated on Gridworld and Reacher and outperforms BC and AIRL.

Strengths

The proposed method is general and applicable to various policy optimization problems. The finding that using distillation is better than simply adding $\ell_1$ regularization to a loss function seems interesting. The experimental results are promising. In particular, transferability to unseen environments is interesting.

Weaknesses

The literature review is insufficient. For example, the structure of EvIL is similar to the framework of learning and evolution. For example, Elfwing et al. (2018) optimize hyperparameters of RL by a simple evolutionary algorithm in the outer loop while the policy is trained by RL. In Nielum et al. (2010), the reward function is optimized by genetic programming. Please discuss the relation between EvIL and these classical studies. In addition, a more detailed survey on meta-learning is needed. For example, Evolved Policy Gradient (Houthooft et al., 2018) optimizes a differentiable loss function, where a similar evolutionary strategy is used in the outer loop. S. Niekum et al. (2010). Genetic Programming for Reward Function Search. IEEE Transactions on Autonomous Mental Development. 2(2): 83-90. S. Elfwing et al. (2018). Online Meta-Learning by Parallel Algorithm Competition. In Proc. of GECCO. R. Houthooft et al. (2018). Evolved Policy Gradients. NeurIPS. My second concern is the scalability of the algorithm. Since a naive evolutionary strategy is adopted to optimize the reward and transition functions, applying the proposed method to problems with a high-dimensional state space is problematic. Finally, it would be better to re-organize Section 5 because Figure 3 is explained after Figure 2.

Questions

1. Figure 1 seems interesting, but I do not fully understand the experimental conditions. The first paragraph of Section 4.1 mentions that the reward function optimized in the training environment is used to train policies in the test environment. However, Figure 1 shows that the reward and transition functions are optimized in the test environment. Please clarify the motivation for the experiments. 2. Figure 1 shows that the performance of EvIL is worse than that of AIRL in the training environment. However, AIRL failed to improve its performance in the test environment. Does it imply that the initial reward is tuned for the training environment? 3. Discussions about Figure 2 are needed. For example, in my view, there are no significant differences among EvIL, EvIL with supervised loss, and EvIL with supervised model. In addition, the performance in the online setting is much better than those of other methods. It implies that EvIL failed to find a good transition function even in the Poitmass environment. Would you discuss the quality of the optimized transition function in detail? 4. Figure 3 is interesting, but I am unsure how the features such as the obstacle position are retrieved from the reward function. As far as I understand, the reward function is represented by a neural network, which is a mapping from a state to a scalar value. Would you clarify the method to retrieve the reward features?

Rating

3: reject, not good enough

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

1 poor

Presentation

2 fair

Contribution

1 poor

Authorsrebuttal2023-11-23

We sincerely appreciate the reviewer's insightful feedback and their thorough understanding of the relevant literature. We are particularly encouraged by the recognition of our approach's generality and wide applicability. Addressing the reviewer's concerns: - While we acknowledge that our current experiments do not exhibit strong scalability, it is important to note that IRL is typically evaluated on MuJoCo environments, which have a smaller observation space compared to some of the environments we employed. Within these environments, we achieved performance parity with the AIRL baseline. - The papers mentioned have been added to the background under the Evolution Strategies subsection (2.1). While Niekum et al. (2010) do utilize evolution strategies to train a reward function, their objective is to enhance RL performance. In our approach, this is not feasible due to the absence of a reward signal. Similarly, Elfwing et al. (2018) also optimize for a reward function, but our framework operates under the assumption that no reward function is available. Evolved Policy Gradient (Houthooft et al., 2018) employs a similar setup with an ES outer loop optimizing for rewards in the inner loop. However, the optimization targets differ between the outer and inner loops in both approaches. To summarise, our approach is fundamentally different for all the papers above as we don’t assume access to a reward function signal we can optimise, but instead we optimise for a cross entropy loss against the expert’s trajectories. Addressing the reviewer's specific questions: - The right-side plots in Figure 1 illustrate how the reward function generalizes to the test environment during optimization on the training data. Notably, the reward function is never optimized on the test data or environment; rather, we solely evaluate its performance there. - The original reward function that both AIRL and EvIL strive to recover remains the same. However, the reward function recovered by EvIL exhibits superior generalization to the test environment. - The plot demonstrates that EvIL, when combined with some knowledge of the reward function (as is often the case in real-world problems), can enhance performance in the offline setting. This is shown in Figure 2a. In Figure 2b, we show the performance of Offline Evil, here the transition function is learned only through Evolution Strategies and a Cross Entropy loss between the trained agent and the expert trajectories. This suggests our ability to recover transition dynamics from the data. In Figure 2d, we show the performance of EvIL where the dynamics model is learnt through supervised loss. Figure 2f, where we show the performance of Online EvIL, is not directly comparable to the other plots, as the optimisation process has access to more information (i.e. the environment). Offline EvIL did not match the performance of the expert (and Online EvIL) as it was only trained on a single expert trajectory. We have additional results on the Reacher environment (with more expert data) that intuitively demonstrate improved performance as the number of expert trajectories increases. The results have been added to the Appendix. - In the experiment depicted in Figure 4, we employ a partially known transition function and aim to uncover the remaining unknown variables. Our results demonstrate the successful recovery of these unknown variables, including obstacle position and size. Therefore it is the transition function, rather than the reward function, that serves as the source of this information.

Reviewer 4scx6/10 · confidence 3/52023-10-31

Summary

This paper presents an evaluation strategy approach for immitation learning. This meta-learning approach optimizes the parameters of a reward function and environment dynamics in the outer loop, and then uses them in the inner loop to reinforcement learn. The authors show that their approach is able to learn the reward function induced by an expert bot behaviour, also when the dynamics of the environment are not fully known. The method is evaluated on gridworld, reacher, and pointmass environment and compared to a standard behavioural cloning approach and adverserial inverse RL.

Strengths

- The paper presents an interesting application of ES, taking advantage of the fact that neither the reward nor the training - procedure need to differentiable. - The approach addresses some key limitations of most IL approaches - Baseline comparisons and ablation tests are sensible - Promising results in three benchmarks

Weaknesses

- Some acronomys are not defined, e.g. AIRL (adverserial inverse reinforcement learning). What are its training setup? - Approach should be evaluated on more environments and potentially complexer ones. Is the approach generally better to BC or does it depend on the type of environment? - It would be useful to test how transferable the learned reward functions are to similar domains - Hyperparaemters for some of the experiment steps don’t seem to be mentioned (e.g. learning rates, etc.?) - The environments are not described in the paper. Details should at least be added to the appendix. Minor comments: Typo page 3: "the the expert demonstration"

Questions

- Why were these particular evaluation test environments chosen? - What are the parameters for the BC training?

Rating

6: marginally above the acceptance threshold

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Authorsrebuttal2023-11-23

We wholeheartedly appreciate the reviewer's encouraging feedback and their recognition of the intriguing application of our approach. We are delighted that the reviewer acknowledged the advantages of utilizing non-differentiable rewards and training methods. Addressing the reviewer's concerns: - We have addressed the reviewer's suggestions by including the BC hyperparameters in the appendix, defining the acronyms, and providing detailed descriptions of the environments. - The transferability and generalizability of the recovered reward function is what we are testing for in the test version of our environment (e.g. different joint dampening in Reacher and a maze vs no-maze setting in the Gridworld) - AIRL stands for Adversarial Inverse Reinforcement Learning, which is essentially equivalent to Generative Adversarial Inverse Reinforcement Learning (GAIL). In AIRL/GAIL, there are two players: a discriminator (used as reward function) and a player. The reward function player tries to pick out differences between the current learner policy and the expert demonstration, while the policy player attempts to maximise this reward function to move closer to expert behaviour. This setup is effectively a GAN. For a more theoretical explanation, please refer to [1]. - We are currently in the process of expanding our method to MuJoCo tasks by changing environment parameters like link lengths and joint stiffness to generate a diverse range of dynamics. This approach generally surpasses BC due to its foundation in Inverse Reinforcement Learning (training the agent in the inner loop) instead of simply imitating the expert's actions. BC has been shown to suffer from compounding errors and poor generalization. To address the reviewer's questions: - The Gridworld and Reacher environments were chosen because they pose a non-trivial challenge in recovering the reward function. The objective in these environments is to change position at the beginning of each episode, unlike other environments like CartPole which rely solely on a positive reward signal. [1] Swamy, Gokul, et al. "Of moments and matching: A game-theoretic framework for closing the imitation gap." International Conference on Machine Learning. PMLR, 2021.

Reviewer vCw23/10 · confidence 4/52023-11-04

Summary

This paper presents a way of applying evolutionary optimization to inverse reinforcement learning. The performance of the method is evaluated in some toy problems and compared with AIRL.

Strengths

It is certain that evolution can provide a solutions to any problem. Originality of the proposal includes: • estimating the gradients of policy and transition model parameters with Gaussian mutation and perform gradient decent, rather than usual selection. • tuning of inner-loop steps to avoid disappearance of the above gradient • L1 distillation of the estimated reward functions

Weaknesses

A crucial issue in evolutionary approach is how practically and competitively a real-world problem can be solved. Figure 1 presents the comparison with AIRL, but the implementation of AIRL is not sufficiently documented. The x axis is the number of outer loops, but how many interactions with the environment happened in each outer loop for EvIL and AIRL? The basic parameters like the population size N should be reported in the main text.

Questions

How does the gradient approach compare with standard selection approach in evolutionary optimization? This can be benchmarked in addition to the ablation study within the EvIL framework. 4.1, 2.: The link between the non-differentiable objectives and potential-based shaping is not clear to me. Page 2, para. 3: environment dynamics are trained: did you mean dynamics models are trained? You can train your agent but not the environment. What were the target reward functions of each benchmark? It is not clear even in the supplementary materials. For each goal point, how sparsely was the reward functions set?

Rating

3: reject, not good enough

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

1 poor

Contribution

1 poor

Authorsrebuttal2023-11-23

We express our gratitude to the reviewer for their valuable feedback and for recognizing the novel contributions of our work, particularly the L1 distillation technique and the dynamic adjustment of the number of inner loop gradient updates to prevent gradient vanishing issues. Addressing the reviewer's concerns: - While the number of inner loop steps for both AIRL and EvIL is the same, in the EvIL case, we leverage the power of parallel computation by training 128 agents simultaneously. This parallel training strategy significantly enhances the computational efficiency of our approach. - To address the reviewer's request, a description of the AIRL implementation, along with the corresponding hyperparameters, has been included in the Appendix. Question responses: - The comparison with gradient approaches is the one we did by comparing with AIRL, the standard approach in this case - Optimizing a potential-based shaping function using conventional meta-learning methods poses a significant challenge. As potential-based shaping only affects training speed, and not the optimal policy, it cannot be optimised with first-order methods. EvIL presents a perfect solution, as it is a zeroth-order optimisation method. The shaping objective could be optimized concurrently with the standard Cross-Entropy objective, resulting in better-shaped reward functions that are faster to optimize. - Yes, we meant the dynamics models are trained, this has been fixed in the original paper - The reward function for the Reacher environment is the standard one available in Gymnax and Brax, which utilizes a continuous reward signal based on the distance between the agent's end effector (“fingertip”) and the goal position. In the Gridworld environment, a reward of 1 is granted if all goals are discovered in the correct sequence before the episode ends; otherwise, the reward is 0. Similarly, in the goal point environment, the reward is only provided when the agent reaches the goal. This description has been added to the paper’s Appendix.

Reviewer p69Z3/10 · confidence 3/52023-11-08

Summary

This paper introduces Evolutionary Imitation Learning (EvIL), a versatile strategy for imitation learning (IL) that can forecast the behavior of agents in environments across different dynamics. EvIL employs Evolution Strategies to concurrently meta-optimize the parameters (such as reward functions and dynamics) that are input into a subsequent reinforcement learning process. The authors claim that they are able to inherit some of the benefits of IRL and In addition, since they are using a population-based method, the reward or the training process does not need to be differentiable. They validate the robustness and generalization ability through some simple environments. The results show that they can perform better than BC and classic IRL approaches.

Strengths

- The ability to predict an expert’s behavior in an environment different from the one where the trajectory data was collected is an important topic - Able to handle undifferentiable rewards.

Weaknesses

- Since the paper only uses experiments to demonstrate the method's effectiveness, I believe that the experiments are too few and too simple. Usually, when someone talks about the generalization of RL regarding the transition function, I will think of more challenging and practical modifications like different gravity, mass, and friction. - The method is very slow since it involves multiple rounds of training. - The authors do not clarify why we should use their design since the method is not the most straightforward way to solve the problem of previous work. An easy way to focus on only one scenario first and talk about why we need the method in that scenario (e.g., online).

Questions

- It seems like in expert demonstrations you only use one kind of transition function is that correct?

Rating

3: reject, not good enough

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

2 fair

Contribution

2 fair

Ethics concerns

N/A

Authorsrebuttal2023-11-23

We sincerely appreciate the reviewer's comprehensive review of our work. Addressing the reviewer's concerns: - We are currently engaged in expanding our method's applicability to MuJoCo tasks, where we intend to introduce variations in environment parameters such as link lengths and joint stiffness to create a diverse range of dynamics. - The iterative nature of our method, involving multiple rounds of training, aligns with standard practice for IRL methods. Both primal IRL approaches like GAIL/AIRL and dual IRL techniques like MaxEnt IRL are iterative procedures. Additionally, our method is not slower than standard IRL approaches, but is comparable due to the massive parallelization we are able to achieve via JAX. As an example, the PureJaxRL library archives a 1000x speed up over other standard PyTorch RL implementations. - As you rightly pointed out, our method's strength lies in its ability to optimize non-differentiable reward functions through a non-differentiable optimization process. Moreover, we emphasize behavioral prediction outside the training distribution and demonstrate our method's superior performance compared to standard AIRL. While AIRL performs satisfactorily within the training environment, our experiments highlight our ability to achieve parity with traditional methods while expanding the range of applicable reward model families. For instance, we can optimize for symbolic expressions of environment features, as demonstrated in the recent EUREKA paper [1]. Additionally, we show better performance in the test environments, indicating rewards recovered by EvIL generalize better. Question response: - Yes, the expert trajectories are indeed sampled from a single environment. [1] Ma, Yecheng Jason, et al. "Eureka: Human-Level Reward Design via Coding Large Language Models." arXiv preprint arXiv:2310.12931 (2023) [2] Lu, Chris Xiaoxuan et al. “Discovered Policy Optimisation.” ArXiv abs/2210.05639 (2022)

Area Chair fMztmeta-review2023-12-14

Meta-review

The paper introduces Evolutionary Imitation Learning (EvIL), for both tabular and continuous control settings. By optimizing the current reward function through reinforcement learning, EvIL uses function approximators such as deep networks to represent the reward function and policy. The authors commit to reproducibility by including hyperparameters in the Appendix and releasing their code on Github. ES methods typically don't go with theoretical analysis. The experimental results are very limited, without sufficient comparisons with representative IL methods. The speed and scalability should also be discussed.

Why not a higher score

No theoretical analysis (kind of acceptable for ES methods). The experimental results are very limited, without sufficient comparisons with representative IL methods. Need more justification on the advantage of proposed method.

Why not a lower score

N/A

© 2026 NYSGPT2525 LLC