Deep reinforcement learning (RL) works impressively in some environments and fails catastrophically in others. Ideally, RL theory should be able to provide an understanding of why this is, i.e. bounds predictive of practical performance. Unfortunately, current theory does not quite have this ability. We compare standard deep RL algorithms to prior sample complexity bounds by introducing a new dataset, BRIDGE. It consists of 155 deterministic MDPs from common deep RL benchmarks, along with their corresponding tabular representations, which enables us to exactly compute instance-dependent bounds. We choose to focus on deterministic environments because they share many interesting properties of stochastic environments, but are easier to analyze. Using BRIDGE, we find that prior bounds do not correlate well with when deep RL succeeds vs. fails, but discover a surprising property that does. When actions with the highest Q-values under the random policy also have the highest Q-values under the optimal policy (i.e. when it is optimal to be greedy on the random policy's Q function), deep RL tends to succeed; when they don't, deep RL tends to fail. We generalize this property into a new complexity measure of an MDP that we call the effective horizon, which roughly corresponds to how many steps of lookahead search would be needed in that MDP in order to identify the next optimal action, when leaf nodes are evaluated with random rollouts. Using BRIDGE, we show that the effective horizon-based bounds are more closely reflective of the empirical performance of PPO and DQN than prior sample complexity bounds across four metrics. We also find that, unlike existing bounds, the effective horizon can predict the effects of using reward shaping or a pre-trained exploration policy. Our code and data are available at https://github.com/cassidylaidlaw/effective-horizon
Paper
Similar papers
Peer review
Summary
The authors aim to improve the theoretical understanding of RL. The contributions of the paper are as follows: 1. The authors curated a dataset called BRIDGE consisting of 155 deterministic MDPs along with their tabular representations 1. The authors found that, for a given deterministic MDP, deep RL is more likely to succeed **if the policy acting greedily w.r.t to the random policy's Q function (which can be obtained via dynamic programming) is optimal**. 1. Motivated by the property in the last bullet point, the authors propose a complexity measurement metric called **effective horizon** that roughly measures the number of steps of look-ahead search needed to identify the next optimal action. * The authors show effective horizon is more reflective of PPO and DQN's empirical performance than prior complexity bound metrics
Strengths
The paper proposes a highly useful benchmark of deterministic MDPs, which is helpful for more theoretical analysis instead of relying merely on results on black-box algorithms. The finding that deep RL is more likely to succeed if the policy acting greedily w.r.t to the random policy's Q function (which can be obtained via dynamic programming) is optimal is extraordinary and quantitatively verified. Furthermore, the effective horizon bound is highly efficient and predictive of the agent's performance, which is useful for future theoretical studies. Last but not least, the paper is well written.
Weaknesses
The paper is well-written, and it is good to see that the authors have released the training code. However, I do have a concern about the PPO training results with Atari. Figure 5 on page 44 shows the training results of Atari with 50M steps (200M frames). However, the scores obtained by PPO are considerably lower than those from popular RL libraries. Below is a table summarizing the performance difference between the results reported in this paper compared to the results reported for `openai/baselines` in the [openrlbenchmark project](https://github.com/openrlbenchmark/openrlbenchmark/blob/8a1248b63daef6b4bab78467a2d7356038b87042/README.md#compare-cleanrls-ppo-with-openaibaseliness-ppo2-on-atari-games). **The PPO in this paper used 5x more samples and produced only 1/3 to 1/15 of the scores of `openai/baselines`.** While the settings are clearly different in some ways (e.g., I noticed the authors did not use frame stacking in the Atari preprocessing), the performance discrepancy is large enough that I think warrants further investigation. | Environment | this paper's RLLib PPO @ 50M steps | `openai/baselines`' PPO @ 10M steps | | --- | --- | --- | | Beam Rider | ~700 | 2835.71 ± 387.92 | | Breakout | ~30 | 405.73 ± 11.47 | | Enduro | ~200 | 986.69 ± 25.28 | | Pong | 20 | 20.45 ± 0.81 | | Qbert | ~1000 | 15228.25 ± 920.95 | | Seaquest | ~400 | 1518.33 ± 400.35 | | Space Invaders | ~200 | 1019.75 ± 49.08 | PPO has many implementation details ([Engstrom, Ilyas, et al., 2020](https://openreview.net/forum?id=r1etN1rtPB); [Andrychowicz, et al., 2021](https://openreview.net/forum?id=nIAxjsniDzg); [Huang et al., 2022](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/)) which may cause subtle issues. For sanity checks, maybe it's worthwhile conducting a PPO RLLib experiment with SB3's [Atari wrappers](https://github.com/DLR-RM/stable-baselines3/blob/master/stable_baselines3/common/atari_wrappers.py) for 10M steps. If the performance is on par with `openai/baselines` PPO results, then the PPO results of this paper can be attributed to different environment preprocessing. If PPO RLLib significantly underperforms, it could mean that PPO RLLib is not a proper PPO implementation, and the PPO results of this paper should be re-done with SB3's PPO, `openai/baselines`' PPO, or CleanRL's PPO, all of which are more well-benchmarked. P.S. According to https://github.com/openrlbenchmark/openrlbenchmark's documentation, you can retrieve the data above by running ```bash pip install openrlbenchmark==0.2.0 wandb login python -m openrlbenchmark.rlops \ --filters '?we=openrlbenchmark&wpn=baselines&ceik=env&cen=exp_name&metric=charts/episodic_return' \ 'baselines-ppo2-cnn?cl=openai/baselines PPO' \ --filters '?we=openrlbenchmark&wpn=sb3&ceik=env&cen=algo&metric=rollout/ep_rew_mean' \ 'ppo?cl=SB3 ppo' \ 'dqn?cl=SB3 dqn' \ --filters '?we=openrlbenchmark&wpn=envpool-atari&ceik=env_id&cen=exp_name&metric=charts/avg_episodic_return' \ 'ppo_atari_envpool_xla_jax_truncation?cl=CleanRL PPO (JAX)' \ --env-ids BeamRiderNoFrameskip-v4 BreakoutNoFrameskip-v4 EnduroNoFrameskip-v4 PongNoFrameskip-v4 QbertNoFrameskip-v4 SeaquestNoFrameskip-v4 SpaceInvadersNoFrameskip-v4 \ --env-ids BeamRiderNoFrameskip-v4 BreakoutNoFrameskip-v4 EnduroNoFrameskip-v4 PongNoFrameskip-v4 QbertNoFrameskip-v4 SeaquestNoFrameskip-v4 SpaceInvadersNoFrameskip-v4 \ --env-ids BeamRider-v5 Breakout-v5 Enduro-v5 Pong-v5 Qbert-v5 Seaquest-v5 SpaceInvaders-v5 \ --no-check-empty-runs \ --pc.ncols 4 \ --pc.ncols-legend 4 \ --rliable \ --rc.score_normalization_method atari \ --rc.normalized_score_threshold 8.0 \ --rc.sample_efficiency_plots \ --rc.sample_efficiency_and_walltime_efficiency_method Median \ --rc.performance_profile_plots \ --rc.aggregate_metrics_plots \ --output-filename figures/0compare \ --scan-history ``` | | openai/baselines PPO | SB3 ppo | SB3 dqn | CleanRL PPO (JAX) | |:----------------------------|:-----------------------|:-------------------|:------------------|:--------------------| | BeamRiderNoFrameskip-v4 | 2835.71 ± 387.92 | 3844.13 ± 266.71 | 5774.61 ± 408.73 | 3133.78 ± 293.02 | | BreakoutNoFrameskip-v4 | 405.73 ± 11.47 | 307.39 ± 118.27 | 251.67 ± 15.92 | 465.90 ± 14.30 | | EnduroNoFrameskip-v4 | 986.69 ± 25.28 | 745.19 ± 139.98 | 493.08 ± 69.14 | 1178.37 ± 90.34 | | PongNoFrameskip-v4 | 20.45 ± 0.81 | 20.69 ± 0.33 | 20.44 ± 0.17 | 20.62 ± 0.18 | | QbertNoFrameskip-v4 | 15228.25 ± 920.95 | 13167.47 ± 1066.11 | 11502.93 ± 487.54 | 17318.50 ± 385.80 | | SeaquestNoFrameskip-v4 | 1518.33 ± 400.35 | 1538.04 ± 389.55 | 2936.30 ± 941.92 | 1203.30 ± 364.07 | | SpaceInvadersNoFrameskip-v4 | 1019.75 ± 49.08 | 911.16 ± 96.51 | 778.57 ± 46.76 | 1164.26 ± 176.96 |
Questions
The finding on the "surprisingly common property" is very interesting to me. I would be interested in more content on it. For example, how did you find this property? What is the motivation for finding the Q values of the random policy? Is there an example of how $\prod(Q^{\pi^{rand}})$ is constructed for a simple MDP?
Rating
8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
4 excellent
Presentation
4 excellent
Contribution
4 excellent
Limitations
Yes.
Summary
This paper proposes a new complexity metric, denoted as effective horizon, for deterministic MDPs. It is defined as the number of full value iteration updates starting from an initial policy, plus a logarithmic factor for concentration bounds, needed to reach an optimal policy. A corresponding algorithm, GORP, is also suitably defined. The bound is calculated for a set of 155 deterministic MDPs, and compared with empirical sample complexity of PPO and DQN, and is shown to correlate decently well compared to other sample complexity bounds. It is also shown how the effective horizon can be applied to shaped rewards or different initial policies to get a better problem-dependent estimate of sample complexity.
Strengths
There are extensive results and analyses in the paper and appendix, with good comparisons to other sample complexity measures. The task of converting many large MDPs to tabular representations and computing bounds can be very useful for future research on tighter theoretical bounds. The proposed measure of effective horizon does seem to have good correlation with empirical algorithms, and intuitively makes sense, as the number of value iteration steps can correlate with the number of gradient updates to a value network. The biggest strength of effective horizon is that it is policy and reward dependent, which makes it much easier to apply (as the paper points out) to shaped rewards and better initialized policies.
Weaknesses
There are many references to tables or figures in the appendix without marking them as being in the appendix, which can be confusing. As mentioned in the paper, there are several limitations such as being only for deterministic MDPs, and hard to compute in general for large MDPs. Nevertheless, the initial results do demonstrate the viability of this type of bound, which future work can take and build upon. Another downside is that the effective horizon can be hard to apply to algorithms with additional exploration built-in, such as using an intrinsic reward. This is due to the fact that the intrinsic reward is non-stationary and changes over time, so the effective horizon would only give a bound for a specific snapshot of intrinsic reward, rather than the entire training process. How to handle non-stationarity could be an interesting future direction. Some concerns about the experiments are brought up in the questions section below.
Questions
How do you define optimal policy across your environments since your episodes are relatively short? Many Atari games go on for thousands if not hundreds of thousands of timesteps, so what does it mean to reach optimality? For the empirical GORP results in table 2, what value of k and m were used? Did you sweep over k to find the best k? Or use the smallest k that reaches optimal? For table 2, for the horizon T, were you using the horizon of the tabular representation (i.e. after frameskip)? If so, there is a degree of arbitrary choice here, as many of the Atari environments had large frameskips of 30, which greatly reduces the T needed to see reward. If frameskips were smaller, such as 4, and horizon increased accordingly to accommodate, then all theoretical bounds would be even more orders of magnitude off from empirical sample complexity.
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
3 good
Presentation
2 fair
Contribution
3 good
Limitations
Are addressed in the paper.
Summary
The paper investigates the question of what finding an explanation for the performance of deep RL algorithms. To accomplish this, the authors propose the notion of effective horizon. It is found that this notion is better able to answer questions about sample complexity and effect of reward shaping or policy initialization in RL than all prior theoretical bounds.
Strengths
Strengths: 1. The first great part about the paper is the writing. Reading the paper is easy and flows well. 2. The authors present a surprising finding - on a large fraction of environment just evaluating random policy with one step of policy improvement leads to optimal policy. For the rest if the environments, a few steps of value iteration usually leads to optimal policy. This is in contrast to previous ideas and theory in RL which are unable to explain when RL fails in certain environments and when it will work. 3. Authors propose a new method GORP (Greedy over random policies), which performs a few step of policy improvements over a random policy and is shown to be a competitor to PPO and DQN algorithms even for long horizon. 4. The experiments demonstrate that the theoretical bounds for GORP neatly explains empirical bounds obtained by DQN/PPO on complex environments by measuring quantities like correlation, medium ratio, etc. 5. The analysis in the paper is also the first to explain the effect of reward shaping and initializing RL with a policy using the GORP bound. The effective horizon bound has a correlation of around 0.4 in both of these settings (reward shaping and policy initialization) which is more than other bounds which are agnostic to reward or do not consider the effect of policy past the horizon.
Weaknesses
Weaknesses: 1. Unlike prior bounds using UCB and EPW, GORP seems to require oracle information to compute the effective horizon. Can the authors comment on the implication of this in the paper? For example, we can’t use GORP bounds to compute sample efficiency without having an algorithm that solves the task first. 2. It is claimed that GORP achieves an optimal policy in almost all the environment when a effective horizon > 1 is used (Line 77), but the experiments for this seem to be missing. Learning curves and final performance along with effective horizon used would be important to support this claim. 3. Novelty of GORP: It would be important to do a more thorough literature review with prior works in the area which proposed an algorithm very similar to GORP in order to clarify the novelty there: a. MCTS b. Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm, Silver et al c. Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model, Schrittwieser et al d. Plan Online, Learn Offline: Efficient Learning and Exploration via Model-Based Control, Lowrey et al e. Learning Off-Policy with Online Planning, Sikchi et al f. Model-Based Offline Planning, Argenson et al
Questions
1. Does GORP with a particular effective horizon outperform DQN/PPO? This experiment seem to be missing in Figure 5 and figure 10 where k is fixed to be 1. 2. How is line 7 in Algorithm 1 performed in practice? Is the whole list of possible actions enumerated to find the best one? If so, does the method assume known model of the environment? In that case, why does comparing to only model-free methods make sense for sample complexity bounds?
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
4 excellent
Presentation
4 excellent
Contribution
4 excellent
Limitations
Authors address the limitation in Section 7 of the paper.
Summary
The paper aims at addressing the gap between RL theory and deep RL practice. The central finding of the paper is a property that correlates well with the performance of PPO and DQN algorithms on a set of 155 environments (such as Atari, Procgen, and MiniGrid). The property looks at whether the argmax action of the action-value function under the random policy coincides with the argmax action of the optimal action-value function. Inspired by the property, the authors propose a complexity measure called the effective horizon that corresponds to the number of lookahead search steps needed to identify an optimal action, given that the leaf nodes are evaluated under the random policy. Bounds that utilize the measure correlate with empirical sample complexity of PPO on the collection of 155 games (r = 0.79 for PPO and r = 0.65 for DQN). Lastly, the paper demonstrates the predictive power of the effective horizon when using reward shaping and pre-training.
Strengths
- The paper is well-written and easy to follow. - Motivation is solid: the paper makes a step certainly in the right direction as the gap between RL theory and practice is tremendous. - The personal experience of the reviewer with the submission started from "That's surprising", followed by "It can't be true", finally reaching "Ah, it makes sense", implying that the paper offers an interesting and non-trivial way to think about RL. One of the take-aways for the reviewer is that if the policy evaluation is done perfectly, there might be a need only for just a few (sometimes zero) policy improvement steps. The reviewer is inclined to recommend the paper for acceptance because it is likely to generate a fruitful discussion in the community.
Weaknesses
There might be certain weaknesses that the reviewer didn't notice but as of now there are none that feel worthwhile mentioning; having said that, the reviewer has limited experience with complexity bounds in RL and maybe more experienced RL theory person will be able to identify the limitations of the analysis. The reviewer is looking forward for the discussion phase.
Questions
Could you quantify the agreement between the (exact) action-value function corresponding to a random policy and a Q-network with randomly initialized parameters? If the agreement is high, it'd somewhat imply that for many domains the weights are initialized at a point that is almost an optimal solution; such a finding would be even more surprising. The reviewer would suggest to take a look at the Policy Churn paper [1] that observes that the argmax of the learned action-value function in DQN-based methods changes in 10% of the states after each parameter update. The observation might be related to the high predictive power of the measure the authors observe. [1] Schaul, Tom, André Barreto, John Quan, and Georg Ostrovski. "The phenomenon of policy churn." Advances in Neural Information Processing Systems 35 (2022): 2537-2549.
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
4 excellent
Presentation
4 excellent
Contribution
3 good
Limitations
Perhaps the main limitation is that the effective horizon could be used for analysis purposes only; from the practical viewpoint, it is unclear how for a large MDP estimate efficiently the effective horizon (and thus obtain the sample complexity for the MDP).
Reviewer response
Thanks for the clarification and answering the questions. It would be great if you can incorporate parts of the response above to make clear (1) differences from model-based methods mentioned above and how you bridge the gap from them to dqn/ppo (2) stress that effective horizon is not just k, possibly giving an intuition for the same.
We will update the paper
Thanks for your response. We will definitely incorporate explanations of both points into the paper. We appreciate the feedback to make our exposition more clear.
Reply to authors
I thanks the author for the detailed reply. > We also plan to rerun our experiments with the SB3 implementations of PPO and DQN to make sure that they are robust to the exact implementations of the RL algorithms. We will include these results in the final paper if accepted. That sounds like a good plan, and thanks for entertaining this thought. I really like the writing and ideas of the paper and think having the SB3 experiments can make the paper more solid. > We believe that the difference in returns is because we limit agents to a single life in the Atari games, as described in Appendix F.1. This is a perfectly reasonable explanation. My reservation is that DRL is brittle, and there is a non-zero chance that 1. you experiment with SB3's PPO and DQN and find the claims are no longer supported by the experiments, or 2. you align the Atari and hyperparameter setting with the RLlib's codebase and find RLlib's PPO still underperforms, which causes a reproducibility issue, making it difficult to attribute and interpret the paper's claims. To give an example of the second issue, [Luo et al., 2020](https://arxiv.org/pdf/1912.00167.pdf) claim their proposed approach IMPACT outperforms IMPALA because their IMPACT gets ~550 score in Breakout whereas their IMPALA baseline gets ~500 score. However, the [original IMPALA](https://arxiv.org/pdf/1802.01561.pdf) reported 640.43 for IMPALA (shallow) and 787.34 for IMPALA (deep). This is clearly a reproducibility issue on their IMPALA, and it's unclear whether IMPACT can still outperform a more aligned IMPALA baseline. > RLlib’s PPO implementation has been benchmarked here (https://docs.ray.io/en/latest/rllib/rllib-algorithms.html#ppo) in several Atari environments and performs quite similarly to SB3, so we do not think that the RLlib implementation is the reason for the difference in results. Thanks for sharing the RLlib's benchmark results. However, the benchmark results are concerning for several reasons. 1. The benchmark results are almost 5 years old (benchmarked on 2018-09 (https://github.com/ray-project/rl-experiments#2018-09)). I could not find a benchmark on their more recent codebase. 2. The benchmark results are not sufficient (only tested on 4 envs); also, it appeared only to use 1 random seed. 3. The benchmark showed RLlib's PPO underperforms `openai/baselines`' PPO, SB3 ppo, and CleanRL PPO (JAX) in **4 out of the 4 tasks** reported. | | openai/baselines PPO | SB3 ppo | CleanRL PPO (JAX) | RLlib's PPO @ 10M steps | |:----------------------------|:-----------------------|:-------------------|:--------------------|:------------------------| | BeamRiderNoFrameskip-v4 | 2835.71 ± 387.92 | 3844.13 ± 266.71 | 3133.78 ± 293.02 | 2807.00 | | BreakoutNoFrameskip-v4 | 405.73 ± 11.47 | 307.39 ± 118.27 | 465.90 ± 14.30 | 104.00 | | EnduroNoFrameskip-v4 | 986.69 ± 25.28 | 745.19 ± 139.98 | 1178.37 ± 90.34 | - | | PongNoFrameskip-v4 | 20.45 ± 0.81 | 20.69 ± 0.33 | 20.62 ± 0.18 | - | | QbertNoFrameskip-v4 | 15228.25 ± 920.95 | 13167.47 ± 1066.11 | 17318.50 ± 385.80 | 11085 | | SeaquestNoFrameskip-v4 | 1518.33 ± 400.35 | 1538.04 ± 389.55 | 1203.30 ± 364.07 | - | | SpaceInvadersNoFrameskip-v4 | 1019.75 ± 49.08 | 911.16 ± 96.51 | 1164.26 ± 176.96 | 671.00 |
Results with SB3 implementations
Thank you for your response to our rebuttal. We agree that deep RL is brittle and that it is important to make sure that our main results are robust to implementation details and hyperparameters. Since the rebuttal period, we have now been able to re-run our main experiments with the Stable Baselines 3 implementations of PPO and DQN. That is, we have trained PPO and DQN agents for 5 million steps across 5 random seeds on all the environments in BRIDGE using the Atari hyperparameters from the `rl-zoo` repository [here](https://github.com/DLR-RM/rl-baselines3-zoo/blob/master/hyperparams). We did tune the hyperparameters somewhat to each environment—we tuned $\text{n\\_steps} \in \\{128, 1280\\}$ for PPO and tuned $\text{exploration\\_fraction} \in \\{0.1, 1\\}$ for DQN. Here are our main results from Table 2 when using the results from the SB3 implementations instead of RLlib: | Bound | PPO: Correl. | Median ratio | AUROC | Acc. | DQN: Correl. | Median ratio | AUROC | Acc. | |----------------------------------------|-------------|----------------------|----------|----------|-------------|----------------------|----------|----------| | Worst-case ($T \lceil A^T / 2 \rceil$) | 0.18 | $1.3 \times 10^{10}$ | 0.62 | 0.61 | 0.18 | $5.5 \times 10^{10}$ | 0.67 | 0.75 | | Covering length ($T L$) | 0.29 | $4.5 \times 10^{6}$ | 0.78 | 0.73 | 0.31 | $3.9 \times 10^{6}$ | 0.81 | 0.83 | | EPW ($T^2 A^W$) | 0.67 | $4.7 \times 10^{4}$ | 0.80 | 0.73 | 0.61 | $8.0 \times 10^{4}$ | 0.87 | 0.84 | | UCB ($S A T$) | 0.22 | **20** | 0.67 | 0.65 | 0.34 | **31** | 0.64 | 0.76 | | Effective horizon ($T^2 A^H$) | **0.80** | **20** | **0.91** | **0.85** | **0.75** | 60 | **0.90** | **0.85** | | *Other algorithm* | 0.75 | 2.0 | 0.84 | 0.83 | 0.75 | 2.0 | 0.74 | 0.98 | | *GORP* | 0.77 | 5.7 | 0.76 | 0.79 | 0.65 | 9.4 | 0.81 | 0.93 | We find that the main conclusions of the paper are unchanged when using the SB3 implementations. The effective horizon-based bounds still correlate the best out of any of the provable bounds with the empirical performance of deep RL algorithms (in fact, the correlation is slightly better with SB3 than RLlib). The UCB bounds are a bit closer in median ratio to the empirical performance of DQN, but the correlation is so poor that we believe the effective horizon remains the best explanation for deep RL's performance. Comparing directly between the RLlib and SB3 implementations of PPO, we found that: * In 37% of environments, the SB3 implementation outperforms the RLlib one (i.e., either the final return is higher or it achieves the optimal return in fewer timesteps). * In 23% of environments, the SB3 implementation performs equally to the RLlib one (i.e., they both achieve the same final return, and if it is optimal, then they achieve it in the same number of timesteps). * In 40% of environments, the SB3 implementation performs worse than the RLlib one. We also found a Spearman correlation of 0.94 between the empirical sample complexities of the two implementations. This suggests that the RLlib and SB3 implementations are relatively similar. We have not yet had the chance to re-run the long-horizon Atari experiments using SB3 (Appendix H.1). However, the fact that we were able to closely replicate our original results using the SB3 implementations—showing that the effective horizon closely correlates with deep RL performance—strongly suggests that our results are robust to the exact implementations of the RL algorithms.
Thank you for your response! I like the paper a lot and my score remains unchanged.
Thank you for the comments! I stick to my score and recommend the paper for acceptance.
Decision
Accept (oral)