Reinforcement Learning with Simple Sequence Priors

Summary

This work proposes to use action-based complexity cost. This work argues that this action-based cost can be formulated either as compression over the experienced so far actions or using a transformer that predicts the next action given the past. This approach is then claimed to provide access to simplified / predictable action sequences. The work is evaluated against a number of dmcontrol tasks and is shown to do better than considered alternatives, both in terms of performance but also in terms of learning policies that are easier to learn. Overall, the work proposes an interesting set of evaluation and presents an interesting idea. However, I found the work fairly difficult to follow at times, whereby paper clarity can be improved. I am also unsure that the proposed solution is necessarily going to be useful, given the presence of theoretically stable alternatives, such as [4]. Finally, I think this work has potential but should be evaluated on more complex tasks - such as tasks from the field of robotics and / or tasks where exploration / initially bootstrapping policy learning is important. As a result, I cannot recommend this work for publication yet.

Strengths

- A novel action sequence - driven reward formulation - Interesting and informative experimentation - Good literature review coverage

Weaknesses

- No complex experiments that can further strengthen the claims for policy search and robustness - Insufficient comparisons to prior works, e.g. such as [4] but also works like HER - Some unclear prose, specifically the abstract, introduction and methods section can benefit from less grandous wording - Additional justification of the choice of transformer (as opposed to something simpler) and also some argumentation as to why this cannot be pre-trained ahead of time and then fixed during training

Questions

Nothing explicit.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The limitations section is a bit lacking, it is unclear how this would work in the context of complex sequential / long-horizon tasks. Also, it is unclear how would this approach work on policies learnt from pixels.

Summary

This paper proposes the idea of using simplicity as a prior when learning control policies using RL. Simplicity (or complexity) here is defined as the cost of predicting action $a_t$ given past actions $(a_\{t - \tau:t-1})$ and current state ($s_t$). Two different approaches are proposed to learn this complexity value. First, using a learned neural network that learns to predict current action given past actions trajectory $(\phi(a_t | a_{t-\tau:t-1}$. This is used as additional reward in SAC. Second approach, aims to use lossless data compression algorithms (lz4) and the motivation is to compute complexity as the additional number of bits required to encode $a_t$ given that we have already encoded $a_{t-\tau:t-1}$.

Strengths

Overall, the paper is well written and accessible. The idea of simplicity as a prior is also interesting.

Weaknesses

I think the paper is very vague in terms of using concepts such as simplicity. In the Related works section, the paper simply connects simplicity with maximum entropy RL and suggests that a uniform policy is simple (”stay close to simple uniform prior policy”). I don’t see this connection, infact I a uniform random policy leads to random walks and infact maybe highly non-simple (see e.g. brownian motion and discontinuous behavior). On the approach side the first approach to predict actions from past is very strange since it results in trying to model non-stationary behavior and thus makes little sense. This is sort of elluded in the paper as well, where the paper suggests that learning a policy with this reward can be challenging. From the results we can further see that this approach really doesn’t work. The idea of using LZ4 would try to reduce new actions thus sort of minimizing surprise. However, interestingly, while this approach performs well it doesn’t perform unifromly well across all tasks e.g. on 5/8 tasks (Figure 5) it performs quite poorly. I also find some baseline SAC numbers to be strange e.g. hopper-hop, quadraped-walk, walker-run have all been shown to be solved using SAC. However, the numbers reported in this paper are very low. I don’t think these baseline numbers are fully accurate or in case there were some other differences maybe the authors can comment. Related Works: There are many other approaches in the literature which focus on simplicity by compressing action sequences into latent variables, or using surprise minimization as additional part of the objective. None of these approaches are referred to or compared in the text. I would recommend the authors to cite them, and compare against them as well. Achiam et al. Surprise-based intrinsic motivation for deep reinforcement learning Berseth et al. SMIRL: Surprise Minimizing Reinforcement Learning in Unstable Environments

Questions

Do the authors know why the baseline numbers (SAC) for these simple environments are very low?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

1 poor

Presentation

2 fair

Contribution

2 fair

Limitations

limitations are discussed in the main paper.

Summary

This paper proposes two approaches to regularizing RL policies based on sequential action priors, rather than on single action priors. One approach is SPAC, which trains an autoregressive open-loop policy prior and uses it as a reference regularizer. Another approach uses the LZ4 compression algorithm to estimate how compressible an action sequence is, and uses that as the regularizer. It tests both approaches along with two baselines, SAC and MIRACLE across a few DM Control tasks, and show that the learned policies end up being simpler (more compressible), as well as more performant.

Strengths

The main strength of the paper lies in the novelty of using LZ4 for a prior and its results. A priori, it is not clear at all whether relying on a dictionary-based compression method that is essentially doing substring matching would translate well to an RL domain, and the results are novel and interesting to see. Including SPAC as well helps to round out the results, showing that a more generic approach to sequence priors is possible. The method and results are mostly quite clear.

Weaknesses

The ablations for noise and open-loop policies are maybe not the most pertinent ablations to have in the main paper. Perhaps there should be ablation for the size and architecture of the transformer prior for SPAC, as a potential way of controlling the complexity/simplicity of the policy prior. It would’ve shed more light on the tradeoff between simplicity vs. performance. Similarly, an ablation for how the actions are compressed with LZ4 (there really should be a short explanation in the main paper about this) such as changing the resolution for discretization would’ve also been very enlightening, to again show the tradeoff between simplicity and performance. Finally it would’ve been very useful to include a different kind of domain, perhaps gridworld-based domains (MinAtar/Atari) to see whether we can see similar trends. ---- After Author Rebuttal ---- I appreciate the additional ablations for the transformer architecture and action discretization. Therefore I have increased my score.

Questions

Did all four agents use the same SAC that was augmented with multiple past actions? Or was the augmented SAC only used for LZ-SAC and SPAC? If this is not the case, then the same augmented SAC should be used for all four agents to be consistent. Robustness to noise ablation: Were the agents retrained with the gaussian noise, or was the noise added to the already-trained agents? The plot shows all agents performance decaying relatively similarly, so it doesn’t seem like LZ-SAC or SPAC were particularly more robust to noise. Perhaps the details of this ablation is better put in the appendix so a different ablation can be in the main paper, such as over the transformer architecture or action discretization for LZ4.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The paper already discusses many of the limitations/weaknesses brought up earlier.

Summary

In this paper, the authors propose a reinforcement learning method that produces simplified action sequences. Simplicity is defined as the predictability of the next action taken by the RL agent and is measured using the number of bits required to encode the action sequence. The authors propose two methods to introduce an information bottleneck for biasing/regularizing the RL agent toward simplicity. The first method SPAC, uses a learned sequence model that predicts the next action and the RL agent is incentivized to produce predictable actions. A second method LZ-SAC directly uses a compression algorithm to compress the action sequence and penalizes the number of bits required for storage. The authors show that this regularization can have several benefits and can lead to better overall performance as compared to other action regularization techniques.

Strengths

The paper is well-written and presented. The motivation provided for the simplicity of action sequences is reasonable and the overall idea is interesting for the community. Several different types of experiments have been carried out, comparing performance, performance per bit of information and the ability to use open-loop control. If validated with more experiments, the proposed LZ-SAC method could be a good way to regularize RL policies. The future ability of RL agents to discard information about the state of the world and simply use prior actions instead is promising.

Weaknesses

The major weakness of the work would be the lack of further baselines. Since the approach is very general, it needs to be validated by comparison with more SOTA off-policy RL baselines to ground the results. It would also be interesting to compare against other information bottleneck methods such as [4]. It would also be beneficial if the authors try to categorize which kinds of environments benefit from compression. It could be that environments without periodicity are worse off with compression. The authors claim that the method never performs worse than SAC but this claim needs to be validated by testing on many more non-periodic tasks.

Questions

- What is the methodology for choosing the compression method LZ4 over others? - Why doesn't any agent do well in the hopper-hop task? In section 7, how much does the performance drop in open-loop control as compared to closed-loop control? It is not clear from the axis in Figure 7. Are the differences in the cumulative rewards between the 4 methods statistically significant? - Figure 10: Why does the LZ-SAC method work the best in this case with stochastic policies, while SPAC performs much worse? Why does stochasticity have this effect?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Yes they are addressed.

Thanks for the detailed rebuttal, well done

Dear authors, I would like to start by thanking you for the detailed and thorough rebuttal to my comments, I am glad you found them useful! I will be happy to increase my score provided I see the referenced by you 1 page long pdf file and the reported updates here match the results there. This document is nowhere to be found, perhaps it is set to different visibility or it is not submitted yet. Thank you once more, great job!

Discussion period summary

Dear all, We would like to express our gratitude to all reviewers for participating in the review process, and engaging with our rebuttal. The reviewers made important suggestions regarding further evaluations and baselines. We addressed these by comparing LZ-SAC to more baselines, like Robust Predictable Control, and by evaluating it on pixel-based DeepMind Control Suite tasks and robotic manipulation tasks from the Metaworld benchmark. We also conducted ablation experiments suggested by the reviewers. Now all reviewers recommend acceptance, with an average rating of 6. We think implementing the changes and extra evaluations suggested by the reviewers made our paper considerably stronger, and we are happy that this is reflected in the increased scores. Thank you all again!