Pitfall of Optimism: Distributional Reinforcement Learning by Randomizing Risk Criterion

Summary

This paper proposes a distributional RL algorithm, PQR, that uses distributional return estimates for exploration by updating with a greedy distributional Bellman operator for time-varying, random risk measures. The paper demonstrates that, under a simple concentration condition on the random risk measures over time, the means of return distributions learned by PQR converge to the optimal Q-values. It is argued that the use of random risk measures (as opposed to static risk measures) produces updates that are less biased away from the optimal Q-values. It is shown that PQR demonstrates favorable exploration performance in a simple chain MDP and in Atari.

Strengths

The paper presents a well-motivated algorithm for leveraging return distributions for exploration in (deep) RL. The idea of using random risk measures in greedy distributional Bellman updates to veer less from the optimal Q-values is novel and interesting. Furthermore, the empirical results of the PQR algorithm appear to be quite good.

Weaknesses

The proposed algorithm itself may not be substantially novel -- I suspect similar results would hold for QR-DQN with a dynamic distortion risk measure that tends to the identity sufficiently quickly. The theoretical results are fairly weak. Firstly, convergence of the return distributions is only established with respect to their first moments. Moreover, the upper bound equation in Theorem 3.3 is difficult to interpret, and I am not sure what one should conclude from this upper bound. Additionally, the influence of the schedule $(\Delta_t)_{t\geq 0}$ and the Dirichlet parameter $\beta$ is not really studied -- I suspect, at least in theory, that there are more optimal data-based schedules. ## Grammatical / non-major issues L4 "present a novel distributional reinforcement learning that selects..." -> "present a novel distributional reinforcement learning **algorithm** that selects..." Last sentence in the abstract cites [6] as a risk-sensitive distributional RL algorithm, but this is not really a distributional RL algorithm. L144 mentions the perturbed probability distribution $\xi\mathbb{P}$, but this doesn't really type-check. $\mathbbf{P}$ is a probability measure (maps measurable sets to probabilities) and $\xi$ is a probability density function (mapping elements of the sample space to reals), so the product $\xi\mathbb{P}$ is not exactly correct. Theorem 3.4 needs to be made more precise. It makes a claim about the fixed point of the Bellman optimality operator, but I think you meant the PDBOO.

Questions

L172 says "PDBOO has a significant distinction in that it performs dynamic programming that adheres to the risk-neutral optimal policy..." -- what does it mean to "adhere to the risk-neutral optimal policy"? The PDBOO updates are still performing greedy updates w.r.t. risk-sensitive policies. As mentioned above, convergence of return distributions is not really established at all under the PDBOO update -- we only know about the convergence of their means. But if the dynamics of the other statistics of the return distributions are essentially unknown, how can you be sure that the random risk measures are actually estimating anything meaningful? Is there any correspondence between the PDBOO targets and the "Thompson Sampling" approach introduced by Riou and Honda (2020) http://proceedings.mlr.press/v117/riou20a/riou20a.pdf? There may be an interesting connection there, which could perhaps help add precision to the statement about "adhering to the risk-neutral optimal policy".

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Limitations are addressed adequately.

Summary

The authors propose a new exploration strategy that is more principled than epsilon-greedy or DLTV of Mavrin et al. (based on truncated variance). The proposed exploration strategy is to based on a novel perturbed distributional Bellman optimality operator, where the goal (Eqn. 1) is to minimize the distributional Bellman objective subject to the perturbation being sampled from an uncertainty set; the uncertainty set is defined as importance weighting the next state's return distribution in the Bellman operator, where the radius of the set decreases with time. The authors show that by appropriately decaying the set's radius, the algorithm can learn the optimal value function, and provide experimental validation on N-chain and Atari, showing competitive performance to IQN and Rainbow.

Strengths

1. Perturbing the expectation with a density ratio is a simple but neat idea. 2. Experimental results seem pretty promising, given that QR-DQN's performance significantly improves with the proposed strategy, sometimes even surpassing IQN.

Weaknesses

1. I am not fully sold that perturbing the Bellman operator is a principled way to modeling epistemic uncertainty of our estimates. In particular, the paper seems to suggest that optimism is actually biased and unprincipled, while we know that in both theory and practice, optimism in the face of uncertainty is principled for online RL and minimax optimal in most well-known settings. 1b. I think the paper would benefit from a clearer and more proper explanation of how prior settings are biased / unbiased, and how the proposed approach is better/worse than prior exploration strategies. 2. if I understand correctly, the proposed uncertainty set seems to be (s,a)-independent, which seems wrong. (in particular, if some parts of the state space was not visited often, that uncertainty should be high, while uncertainty should be low for frequented states).

Questions

1. Typically, distributionally robust optimization (DRO) optimizes the worst-case performance. However, in eqn. 1, you sample from the uncertainty set instead of taking a supremum. Can you give some intuition on why you care about the average performance over the set, rather than worst case? 2. Is the perturbation gap related/inspired by the dual form of dynamic risk with coherent risk measures? 3. Theorem 3.3 shows some kind of convergence of the learned $E Z^{(n)}$ to the true $Q*$ , but it seems that as n -> infinity, the bound does not necessarily to go zero. Doesn't this mean there will be some bias in the proposed approach? 4. To be clear, does your implementation only use QR-DQN + the proposed PQR exploration strategy? Do you make use of target networks, double Q-learning, prioritized replay, or any of the bag of tricks that Rainbow uses? 5. In Section 4.2, what is meant by "intrinsic uncertainty" of Atari? Without sticky actions, Atari is fully deterministic and there is no aleatoric uncertainty (only epistemic uncertainty), right?

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

Please see weakness/questions.

Summary

This paper investigates the exploration problem in distributional reinforcement learning (DRL). It proposes the Perturbed Distributional Bellman Operator (PDBOO) as an extension of the distributional Bellman operator, which introduces non-directional noise to the target return distribution. This extension originates from the problem that a one-sided risk tendency can cause biased action selection, which might lead to unsuitable behavior depending on tasks. To address this issue, the PDBOO adds non-directional noise as a perturbation to the return distribution targeted for learning, ensuring risk neutrality. The perturbation term is sampled at each time step. This approach allows for the selection of diverse actions while maintaining risk neutrality. The authors provide a theoretical analysis of PDBOO, demonstrating how the strength of the perturbation should be scheduled to asymptotically converge to the unique fixed point of the Bellman optimality equation. Then, the PDBOO is applied to the Quantile Regression Deep Q Network (QR-DQN) to propose a method called Perturbed Quantile Regression (PQR). The effectiveness of PQR is evaluated in a 4-states chain MDP and 55 Atari games and compared with several DQN variants.

Strengths

- The paper introduces the Perturbed Distributional Bellman Operator (PDBOO), a novel extension of the distributional Bellman operator. This approach enables diverse exploration while maintaining risk neutrality in distributional reinforcement learning. The authors also develop Perturbed Quantile Regression (PQR), a DQN variant that estimates quantiles based on PDBOO. The proposal represents a new and interesting direction of the study in distributional RL. - The authors provide a solid theoretical analysis of PDBOO, demonstrating how the strength of the perturbation should be scheduled to achieve the unique fixed point of the Bellman optimality equation asymptotically. This result provides confidence in their proposed methods and valuable insights for algorithm development. - The proposed PQR method is evaluated in a 4-states chain MDP and 55 Atari games. While the results showed room for improvement, the evaluation confirms the potential practical applicability of PQR and its ability to address the exploration problem while maintaining risk neutrality.

Weaknesses

- The paper lacks an explanation as to why the proposed approach is more effective compared to other methods. It is crucial to discuss under what tasks PDBOO is effective and when it might not be suitable to understand its value. - For example, PDBOO would add random noise (perturbation) to the target return distribution for learning. Depending on the perspective, this could also have the side effect of making it more challenging to learn the return distribution due to the existence of noise. I would like to know if there are situations where PDBOO should not be applied. - Also, it is not clear why the approach of PDBOO is necessary, as it seems that what the authors want to achieve with PDBOO would not be the perturbation of the target return distribution for learning but instead could also be accomplished by perturbing the estimated return distribution at the time of action selection. This is somewhat supported by the results of p-DLTV in the N-chain MDP experiment. - The overall experimental results seem weak and do not resolve the abovementioned questions. Furthermore, the improved performance of PQR is observed in a subset of games, and the improvement seems modest. There is no comparison with p-DLTV, which showed similar good performance in the N-chain MDP experiment.

Questions

- How exactly does the proposed PQR select actions? Does it use an epsilon-greedy or softmax policy? More importantly, how does it compute the values of the actions which will be the input to the policy? Does it sample the perturbation \xi, as PDBOO does, and skew the distribution for calculating the mean? I think there are two timings at which the distribution can be perturbed by introducing ξ: during the parameter θ update, and during the action selection. This paper only seems to discuss the former, but considering that p-DLTV performs well similarly to the proposed method, wouldn't it also be valid to perturb the mean for the action selection? - What is the type of policy for each baseline used in the experiments? Is it epsilon-greedy or softmax policy? I do not see it stated. - Regarding the N-Chain with high intrinsic uncertainty experiment, the difference in the decaying rate of \Delta and c between PQR and DLTV or p-DLTV might also be significant. What are the authors' thoughts? It would be best to align the experiment conditions as much as possible. - The authors used the N-chain task where risk-based exploration like DLTV obviously does not work well, so the presented results are reasonable. On the other hand, in a task where DLTV works well (e.g., both s_0 and s_4 have the same reward variance), I think it is essential to experiment to see how the proposed method performs to compare the proposed method fairly.

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

2 fair

Contribution

2 fair

Limitations

- The paper lacks a thorough discussion of the limitations of the proposed PDBOO and PQR. The authors should provide more insight into scenarios where the method might not be applicable or efficient. - As pointed out above, why the proposed method is effective (for instance, as opposed to p-DLTV or simply increasing the temperature during action selection) remains unclear. It seems that the questions are not adequately addressed in the experiments. The authors should consider addressing these points to provide a more complete view of their method.

Summary

In this work, the authors address the issue of biased exploration caused by a one-sided tendency on risk in action selection by proposing the method perturbed quantile regression (PQR). PQR selects actions by randoming risk criterion while retaining a risk-neutral objective. The authors also derive a sufficient condition for the convergence of the proposed Bellman operator without satisfying the conventional contraction property. Results are demonstrated in an extended N-Chain environment and the Atari suite showing improved performance.

Strengths

The paper was written very well in a pedagogical manner which provided intuition and understanding of the challenges in the field surrounding the contribution of the method. Deep technical detail was provided with accessible explanations. The motivation for the method was well grounded intuitively. The experiment section also directly confirmed the intended contributions, organizing each experiment and result around specific questions directly aligned to the motivation. The background section was thorough and clear. Since the authors pulled together several topics which many researchers do not study all of, this presentation was very important and added a clear path of accessibility to the technical detail in the rest of the paper. The supplementary material provided sufficient information to reproduce results including code. The authors also included substantial supporting theoretical information in proofs and a large amount of additional convincing experiments. The contribution to the community regarding managing risk in exploration strategies is important and interesting.

Weaknesses

The authors claim limitations about epistemic and parametric uncertainty metrics as well as optimism in the face of uncertainty approaches which I thought were not backed up in the paper strongly enough given that these claims are the premise of the entire work. The paragraph in the introduction which presents the limitations describes it as a broad issue, yet only one paper is cited as an example. To provide more context, it would help to give at least more citations here to back up the importance of this challenge. The deeper explanation of the DLTV paper is not necessary for all additional citations. Please provide more comments in Algorithm 1 (like the current text “Select greedy action…”). The algorithm is fairly clear but with the density of notation, recalling the variable references would be much easier for the reader with textual reminders in the algorithm block. My primary concerns regarding this work are related to the experiments in Section 4.1 which require clarification. Figure 4 is quite hard to read since the text is very small. Reformatting could make these plots readable when the paper is printed. Furthermore, the results in Figure 4 are very hard to understand due not only to the legibility but also due to insufficient explanation in the experimental results section and figure. See Questions for more information.

Questions

I am unsure if I am correctly interpreting the results in Figure 4. Is the lower standard deviation around the dotted line for the mean a1 considered to be a better estimation result? Given that this experimental setting is new (the authors indicate that they adapt the N-Chain environment and cite work - DQN - which presents very different result metrics), substantially clearer explanation is required about the connection between the presentation and the claims.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The paper does not discuss limitations or broader impact. This discussion should be added.

Summary

This paper addresses exploration via distributional RL which has, to date, been most recently performed under an _optimism in the face of uncertainty_ (OFU) paradigm. This paper loosely characterizes this OFU approach as problematic as it results in biased exploration, resulting in sub-optimal value distributions. To remedy this pitfall, this paper proposes a stochastic perturbation of the distributional Bellman operator. Clear theoretical development of the proposed operator gives way to practical algorithm development building on top of quantile regression. Empirically, the proposed algorithm PQR is demonstrated to have clear performance benefits.

Strengths

The paper sets up a really interesting juxaposition between the advantages of the distRL paradigm in accounting for aleatoric uncertainty but attempting to resolve epistemic uncertainty through exploration. As set out in the introduction of the paper, this presents an appealing motivation to perhaps determine more effective exploration strategies that are cognizant of this apparent mismatch. The methods presented in this paper to address the one-sided tendency on risk, resulting in a practical algorithm, are reasonable and seem to neatly satisfy the intended contributions set out in the paper. I was impressed at the extent by which the authors provide sufficient background in distributional RL in Section 2. Many papers take shortcuts here however this paper lays out relevant concepts from which the remaining development of the methods introduced in the paper are easier to understand from this grounding. The technical formulation of the PDBOO is clear and well written. The definitions help to outline important concepts that motivate the further development of the methods. I appreciate how the authors approach Section 4, clearly laying out the objectives of their empirical study. This helps to frame the impressive experimental performance achieved by the introduced PQR algorithm.

Weaknesses

It’s not clear what is meant by “without losing the risk-neutral objective” (mentioned in the abstract)… In line 43, it could be helpful to again list out what the two types of uncertainty are to more directly connect this sentence to what has been introduced conceptually earlier in the paper. The authors seem to use “deep” RL interchangably with “distributional” RL in the introduction. This makes the framing of the work a little challenging to follow initially. Better clarification of prior works in the non-distributional deep RL from distributional deep RL would help make this easier to understand and would strengthen the beginning sections of the paper. The statements made in lines 50-53 are pretty strong, as in they seem to be a result of some analysis or in the worst case are derived from opinionated intuition. Some additional justification with formal explanation (whether through a toy example or reference to later section or citation to a paper) would help make the writing easier to accept and be persuaded by. This is especially important because the “one-sided tendency on risk” appears to be a major foundation for the proposed PDBOO. It would be helpful if the authors expanded more on their discussion at the end of Section 2.2 since DLTV appears to introduce biased exploration that is addressed in this work. A couple of sentences addressing the limitations of DLTV and how the work presented in this paper addresses those would be great. The sentence that ends the preceding paragraph (Line 100-101) is a decent example of setting up a clear understanding of the contributions set forth in the paper. Many of the claims about why PQR performs well on n-chain and in Atari are speculative. The paper would be significantly improved if more rigorous analysis connecting the theoretical treatise in Sections 3.2 and 3.3 to empirical performance and how it differs from the baselines would be great. The discussion around the empirical results is somewhat disappointing. It would additionally be interesting to see any ablations that could be done on PQR to help tease apart the effect of the different components of the algorithm have on performance. I suppose that this is is partially addressed with the hyperparameter plots in Figure 6 but I think it would be helpful to see how sensitive the resulting policy is to the scope of the ambiguity sets used.

Questions

Is the guarantee of risk-neutrality found in expectation? E.g. is it a response to evenly choosing a risk-averse obj vs a risk-seeking obj? The writing in lines 41-45 is pretty difficult to follow. It’s apparent that the authors are taking care to concisely convey their thoughts but fail to do so clearly. Line 46: It’s not clear what “side-effect” is being referred to here. Additionally, it would be helpful to perhaps directly reference what section in the paper is being referred to in this sentence as the authors indicate that they’ve explored the effects of OFU in deep RL and the effects it has on handling the two types of uncertainty. Line 61: Could you call PQR a “stochastic optimization” approach to help situate the work among other prior work? This isn’t a major concern of mine. I’m just thinking that it could be an easier way to describe the approach rather than calling it a “randomized” approach. Line 93: The loss function here is simply the Huber loss, no? It’s not overly important to include this much detail to be honest unless the paper heavily develops further insight directly from these equations (applies to Section 2 in it’s entirety). Otherwise this just feels like a regurgitation of information that’s already established in the prior literature. Line 128-129: Why is it important to have an interpretation of different risk levels as constructing an options framework? How does this help build the proposed methods introduced in this paper? Line 178: Does “the standard” here mean the standard distributional Bellman operator? Line 186: What is $B_\xi$? How does this relate to the symmetric Dirichlet distribution used to construct each ambiguity set? This seems to be addressed in Lines 210-212.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

3 good

Contribution

3 good

Limitations

Several technical statements that crucial to the dvelopment of the proposed methods/contributions are vague and are not precisely justified. For example, it would be really helpful (and would help me agree/accept the claims) in Lines 170-174 if it were shown precisely how Eqt 1 maintains risk-neutral policy performance. As currently written we have to take the authors’ word. Additionally, the statement of using a min-expectation vs. a min-max operator should be developed and justified. This is a new concept that hasn’t been discussed previously. It’s not clear how the time-varying amiguity sets are decoupled from a changing estimate of the return distribution Z. As formulated, PDBOO seems to ignore that $Z_\theta$ is changing through time. The paper omits at least one relevant paper, Moskovitz, et al (NeurIPS 2021) where an ensemble of distributional crtiics are used to balance between optimism and pessmism for continous control. It would be informative to hear from the authors why this paper, and its corresponding approach, should or shouldn’t be included as a comparative baseline. >Moskovitz, Ted, et al. "Tactical optimism and pessimism for deep reinforcement learning." Advances in Neural Information Processing Systems 34 (2021): 12849-12863.

Thank you for the detailed responses

I appreciate the detailed responses from the authors. I am satisfied by the answers which have addressed my major concerns and correcting some areas that I misunderstood. As a result, I have decided to raise my score from 6 to 7. Based on the responses and demonstrated efforts to revise the paper (as recommended by all reviewers) I am confident that the promised changes will result in a stronger paper, one suitable for publication.

Additional Response by Authors

## No comparison with p-DLTV We apologize for the delay in writing about this issue due to the character limit. First of all, p-DLTV is an exemplary algorithm for us to compare OFU and randomized approach very effectively. Also, PQR is a more general method compared to p-DLTV which only considers 2nd order moments (variance), whereas PQR uses all the moments indirectly. We experiment with p-DLTV for some Atari environments in Appendix D.3, but do not include it in the main text for three reasons. 1. Reproducibility problem with DLTV: As we wrote in detail in Appendix D, DLTV was difficult to check for reproducibility because it did not provide raw scores. In Table 4, our implementation showed that DLTV had a Human Normalized Mean/Median Score of 603%/109% based on 40M frames, which is a marginal performance difference compared to QR-DQN's 505%/120%. Therefore, we decided to baseline only those algorithms that were reproducible, and DLTV and p-DLTV were excluded from the baseline for a fair comparison. 2. Hyperparameter sensitivity: While experimenting on several Atari games, we found that p-DLTV is very sensitive to its coefficient, c, as shown in Figure 6. As noted in Appendix C.1, we ran a grid search for six values of c, but this suffered from the problem that the optimal value was different for different environments depending on the scale of the reward, making it less effective. PQR, on the other hand, has the advantage of being intrinsically tunable, since xi is defined as a weight that is independent of scale, and thus showed robust performance. 3. Not consistent with our proposed theory: From a theoretical perspective, p-DLTV does not satisfy the sufficient conditions we proposed. Since we want to guarantee risk-neutrality while the agent explores, p-DLTV is only an intermediate algorithm so it is not the final goal. Furthermore, we experimented further in the global response with PQR-$\sqrt{\log t/t }$, which has the same form of scheduling as p-DLTV and found that it easily falls into suboptimality.