A Method for Evaluating Hyperparameter Sensitivity in Reinforcement Learning

The performance of modern reinforcement learning algorithms critically relies on tuning ever-increasing numbers of hyperparameters. Often, small changes in a hyperparameter can lead to drastic changes in performance, and different environments require very different hyperparameter settings to achieve state-of-the-art performance reported in the literature. We currently lack a scalable and widely accepted approach to characterizing these complex interactions. This work proposes a new empirical methodology for studying, comparing, and quantifying the sensitivity of an algorithm's performance to hyperparameter tuning for a given set of environments. We then demonstrate the utility of this methodology by assessing the hyperparameter sensitivity of several commonly used normalization variants of PPO. The results suggest that several algorithmic performance improvements may, in fact, be a result of an increased reliance on hyperparameter tuning.

Paper

References (21)

Scroll for more · 9 remaining

Similar papers

Peer review

Reviewer ydma4/10 · confidence 4/52024-06-16

Summary

The authors define a environment-parameter sensitivity metric in terms of a Jensen gap of min-max scaled performance. The gap gives us the difference between the best average performance and the average best performance over a particular algorithms hyperparameters over a set of environments. The authors then provide a 2D visualization where they plot this sensitivity gap against the average best-performance. This gives an intuitive way to classify whether a change in parametersensitivity can be outweighed by its (positive) change in performance. The authors run experiments for PPO on brax to illustrate their method. At the end, the authors also use their performance metric to tune subsets of the total parameter set.

Strengths

The definition for the environmnent-dependent parameter sensitivity is intuitive through its use of Jensen's gap. I agree that this is a sound method to evaluate robustness of parameter-tuning. Although, care must be taken when choosing the environment-sets. Figure 3 and the discussion in 3.2 is excellently presented, and gives an intuitive way to think about what patterns we would like to see when performing parameter ablations. The experimental setup is repeated over many repetitions and use appropriate statistical methods, when discussing the results, the authors discuss overlapping intervals and are decently careful about making conclusions.

Weaknesses

### Summary Although this paper shows a nice way to visualize robustness of parameter optimization methods across sets of environments, the severe computation requirements needed to create these visualizations on top of the shallow experimental setup that the authors performed, makes it unlikely that the current results will prove useful to other practitioners (aside from the isolated case applying PPO to Brax). The chosen metric for performance (AUC) is incomplete when considering how RL is often applied (when only caring about final performance), and arguably estimated using a non-robust bound estimator. Furthermore, the authors only test on 5 Brax mujoco environments when comparing extensions that originate from Atari, DMLab, Minecraft or elsewhere. The authors, do not take characteristics of the environments they test on into account when discussing results. As a consequence, the conclusions we can take from the presented results are quite limited. Therefore I vote for reject. ### Major comments - The single choice of AUC metric is poorly motivated, although it is a relevant metric, in RL we are also often only interested in **final performance**. So, the AUC gives a proxy for cumulative regret, I also want to see a proxy for simple regret, the best-run within a algorithm's training cycle. - The authors argue that we need robust estimators of performance for hyperparameter sensitivity. So, I am worried about the choice for min-max normalization of the algorithms' performance, see Eq. 1). These can be extremely high-variance estimators for the performance bounds, and therefore, aren't really robust (perhaps explaining why the authors needed 200 seeds per ablation)... I understand that this always nicely scales results into $[0, 1]$, however, if we rerun the experiments (different seeds and parameter-sets), would Figures 1 and 2 give similar patterns? Would this change our conclusion about what choice of hyperparameter is better in terms of environment robustness? Why not choose for an interquantile range or even a $[p, 1-p]$ percentile range that has guaranteed lower asymptotic variance? - The authors don't discuss the sensitivity/ robustness to the choice of environment sets for Eq.2). I.e., what characteristics do the environments have that we test robustness for. For this reason, I also think that the discussion in 4.3 is a bit shallow considering the choice of Mujoco environments. The rewards in Brax are relatively well scaled and behaved (at least compared to e.g., a game like 2048 which has exponential rewards), so concluding that the value target symlog is not beneficial (on top of my comment about min-max scaling) should be nuanced and discussed appropriately. - It's not clear from Eq.5) (and the whole of section 5) how we decide what reference point to take when deciding which parameters to tune. I.e., when we choose to keep some parameters fixed, what should their values be? Are the defaults by definition robust or not? ### Technical comments - Definition of normalized environment score $\Gamma$ is wrong, why is the action-space $A$ used here? (see section 2). Why is the symbol for environment sets not consistent, why use $E$ instead of $\mathcal{E}$. In section 5 suddenly $\Omega$ is used again in place of $A$. ### Minor comments - Eq. 2) Why not write the sum as an expectation, $\mathbb{E}[\max_h \Gamma(...) | e \sim {p(e)}] - \max_h \mathbb{E}_{p(e)} \Gamma(...)$. I.e., the more recognizable Jensen gap. - Shortly discuss expected behaviour of Eq.2 and what it means, i.e., a value of 1 means a large gap, meaning not robust, and a value of 0 means no gap, meaning very robust. - Line 220, minibatch advantage normalization is not that important for PPO, even the blog-post that the authors cite here says so. - What ranges for the parameters were chosen in section 4.1 ? It's easy to break an algorithm through a bad or uninformed choice for the learning rates... It's also not really clear from the text how the authors created the algorithm configurations to test, i.e., random-search? an exhaustive grid? (Found this in Appendix D, the ranges for the values are OK when speaking from experience, and the authors used exhaustive grid-search; this **must** be included in the main text) - Line 291, "It may ... be gained." This sentence is broken, maybe rephrase: "By finetuning a few important parameters for each specific environment, you can unlock most of an algorithm's best performance."

Questions

- Could the authors change the bound calculation (e.g., IQR or something similar), this should be an easy local change, and add the best-performing run metric (i.e., have the AUC and the best-run per algorithm side-by-side). - Could the authors improve on the discussion and conclusions of the results, discussing the characteristics of the environment and how the choice of parameters hypothetically covaries with this.

Rating

4

Confidence

4

Soundness

2

Presentation

3

Contribution

3

Limitations

Section 6 shortly discuss limitations, however, the actual limitations are discussed in appendix D. The shallow experimental setup that was performed resulted in 7 GPU years on modern NVIDIA GPUs. Since most code was implemented in Jax and the algorithm used was PPO, one of the most light-weight RL algorithms currently out there, the current framework does not offer much headroom for testing other methods. The severe computation costs are not included in the broader impact statement. The authors do state that reducing carbon footprint of DRL experiments are important, however, the presented methodology is not carbon friendly. If, the authors presented a better experimental setup, then this would alleviate the need for other practitioners from running these types of sensitivity analyses for PPO. However, this is currently not the case. Another way to improve the current setup is to use variance reduction methods in some way to reduce the immense compute requirements.

Reviewer aSZK7/10 · confidence 4/52024-06-26

Summary

The paper proposes a method to analyse how sensitive are RL methods with respect to hyperparameter tuning. The author argue that one method may perform well on average but require more HPO tuning per task which hides some computation and prevent from having comparable results. They introduce a sensitivity metric which measures the difference between the best hyperparameter tuned per task and the performance of the best hyperparameter on average. Then, they propose a quadrant analysis where both the sensitivity and the performance of a tuned score are displayed which allows to compare algorithms on those two dimensions. Experiments are then performed on PPO normalization variants where they show that current normalization have different trade-offs: some approaches improves the scores but increase hyperparameter sensitivity while other conversely lower scores and sensitivity but no approach allows to improves both scores and sensitivity currently. Finally, the authors study how many hyperparameters require tuning while having the other fix and while keeping 95% of the best performance.

Strengths

* The paper is very well motivated. It tackles an important problem for RL (and in general on how to account for HPO sensitivity when reporting results. * The paper is very well written and easy to follow, the methods are well described and the experiment are very sound * The paper would provide a valuable contribution if the results of the runs are released (will they?, see my question)

Weaknesses

The paper is currently missing some analysis regarding the method stability (see questions). For instance how much the proposed method would be stable and reliable with respect to different environments (does adding one environment change the results completely?) or normalization (e.g. using CDF instead of min max).

Questions

Here are points that would be important for me to raise my score: * In figure 4, are the results stable if you leave one out one environment? If you compute the plot 5 times each time leaving one of the environment, do you have similar plots? If not, then the analysis on the paper will be less warranted (that such and such method is more stable with hyperparameter). * Will the paper release the dataset of evaluation in addition to the code? (the code would have limited interest compared to the data) I would highly encourage to share the dataset, possibly with a script to reproduce some of the paper figures (for instance as was done by https://github.com/google-research/rliable) The dataset would be also very useful to simulate and compare HPO methods, in particular if it contains the evaluation per iteration (and per 100s of iterations for instance). * Figure 4: there are very large outliers for the obs zero mean normalization sensitivity and the distribution is very skewed, could you explain why they happen? * One popular method is f-Anova to study hyperparameter importance, could you include an analysis using it? It would be useful for practitioners to know which HP are most useful. * In the limitation section, you mention that the results may change under a different normalization (say CDF instead of min-max). This is indeed an important point, could you report the result for Fig4 as well with CDF? I assume it should be a minor change as it is just changing the metric. However, it seems important to assess how much the method would be impacted by different normalization (even if the results change, the paper would still be valuable) Here are points that are more details: - You have l110 p(w, e, h ,\kappa) and then p(a, e, h), it would be nice to unify the notation - l151: I think it would be useful to precise that the quantity is always >= 0 - "The shaded region is a 95% Student t-distribution confidence interval around the mean return over 200 runs " => what is the CI not covering the mean in walker 2D?

Rating

7

Confidence

4

Soundness

2

Presentation

3

Contribution

3

Limitations

Yes

Reviewer yuTN4/10 · confidence 3/52024-06-29

Summary

This paper proposes a new evaluation regime for reinforcement learning. As opposed to only taking into account benchmark performance (i.e., final return), as is ocmmon in previous literature, this work suggests considering an extra dimension of how sensitive algorithms are to hyperparameters in tuning based on a heuristic developed in the paper. They analyse PPO, and some of its variants, using this new evaluation regime, and find that performance increases often correspond to increased sensitivity to hyperparameters. Finally, the paper considers a top-k approach to hyperparameter tuning, adjusting only the most impactful hyperparameter, to see how the baseline algorithms compare under a more limited hyperparameter tuning regime.

Strengths

- The paper clearly takes care to use an extensive number of experiments, possibly reaching into the realm of unnecessary, to provide some rigour to their results. - The domain being considered - considering the hyperparameter sensitivity of algorithms in addition to their performance - is an underexplored area and one which becomes increasingly important as the cost of experiments increase. - Approaching this problem in a visual setting seems reasonable. - I like the approach of distilling hyperparameter optimisation to optimising only a smaller set of more important values; this has real benefits in enabling tuning of the majority of hyperparameters in *cheap* environments and tuning the key values only in more expensive environments.

Weaknesses

- Noting appendix A (the table of how many hyperparameters each algorithm has); the definition of 'hyperparameters' seems pretty weak, and a lot of those included seem to just be design decisions of the actual algorithms. - I find reading the plots quite confusing, exacerbated by all of the different colours marking each of the areas. I think this obfuscates the message of the plots and as a reader makes it hard to come to conclusions. - Given this work is based on PurejaxRL, which I think runs on Brax for 1e8 frames, these experiments seem very short (3e6 frames) and possibly doesn't give the algorithms the full opportunity to converge. - There is very limited discussion of preexisting literature in this space. While I appreciate this takes a subtly different tack (calling for us to measure how sensitive to hyperparameters RL algorithms are, rather than designing algorithms with few hyperparameters), I would expect to see significant more discussion of AutoRL literature. Framing this work better in related work would definitely strengthen the paper. - The formatting looks quite off with the figures. I think this is because a lot of the captions are to the side of the figures, rather than below. - Considering the actual hyperparameters being tuned, the results feels slightly disingenuous. If we assume the practitioner running the hyperparameter tuning, we would possibly expect them to select more reasonable values or focus the search in a significantly more targeted way. For instance, it is no surprise that the highly performant methods saw large performance decreases (i.e., hyperparameter sensitivity) when evaluated with entropy coefficients in the range [0.001, 10] or learning rates spanning 5 orders of magnitude. Instead, it would be much more sensible to explore how the performance changes with reasonable hyperparameters that one is more likely to practically tune over (i.e., we generally have a good idea of where to start with values, so are unlikely to try an entropy coefficient of 10). - It would be good to see some exploration of algorithms which have done something different than just considering the changes to normalisation in PPO returns or observations. It feels like some broad statements are made despite the fact that this analysis was only taken in a single setting.

Questions

- In the plots, the authors state that they plot 95% confidence intervals over 200 seeds; but in the y-axis, all of the 'confidence intervals' are only negative. I am not particularly clear why. Is this a mistake, or am I misinterpreting results? At the same time, are the circles representing the mean score over runs, environments, or both; and therefore, are the confidence intervals defined over the runs, the enviroments or a big stack of (in this case) 1000 results per point? Is comparison for the hyperparameter sensitivity done per seed or for the averages over seeds? I think there's a lot of questions about how exactly these results are being presented which doesn't come through in the paper. - In some cases, increased hyperparameter sensitivity can be a positive - it can give us extra opportunities to boost performance. Do you think this should be mentioned as a limitation of this method, which effectively calls for less sensitivity? - Why did you use 200 seeds per hyperparameter tuning configuration, which is significantly greater than would be used in most other research? In practice, would it not have been more sensible to have used a much smaller number of seeds (eg 8), and focused on a much tighter range of hyperparameters? - As a brief aside, the authors have not removed the instructions of the Neurips checklist as requested, which should be done if the camera ready copy is released.

Rating

4

Confidence

3

Soundness

2

Presentation

2

Contribution

2

Limitations

The authors briefly discuss some limitations of their methods; in particular, that their proposed metrics are highly environment specific (even in this case, they are looking at only one suite of environments which may end up biasing results), and will need reevaluation every time you shift to a new set of domains. However, I think there are some other key limitations that are not discussed, many of which have been raised above. In particular: - In addition to being environment specific, the metrics proposed are going to be very specific to the range of values which are tested. I have mentioned above how the values chosen for hyperparameter tuning here are not particularly reasonable.

Reviewer 6Bue7/10 · confidence 4/52024-07-01

Summary

The paper introduces an empirical framework for assessing the hyperparameter sensitivity of reinforcement learning algorithms. The framework consists of two metrics: 1. hyperparameter sensitivity, which gives a normalized difference in performance between the per-task best hyperparameters and the across-task best hyperparameters. 2. effective hyperparameter dimensionality, which gives the number of hyperparameters that can be left the same as the across-task hyperparameters while tuning the rest and still obtain a threshold say (95%) of the per-task best hyperparameter performance. Using these metrics in addition to performance metrics, practitioners and researchers can have a better idea of the benefits/downsides of a modification to an algorithm (how much better vs how much more sensitive) The framework is used to compare several normalization variants of PPO introduced in the past years on the continuous control environments, giving a better picture of their contribution.

Strengths

The problem is well-motivated. Hyperparameter sensitivity is a well-known issue in deep RL with novel algorithms often providing better performance at the expense of a higher sensitivity. Such tradeoff must be made explicit. The framework introduced in the paper provides an effective way to draw that tradeoff with a clear interpretation of it using Figure 3. The metrics are simple and quite natural and provide a solid starting point for a hyperparameter sensitivity framework. Although computationally expensive and still to prove if its results transfer across domains, the framework is likely to have a high impact on the field. At least on the continuous control with Brax and PPO variants where the framework has been used as an example. In particular thanks to the thorough and accurate experimental protocol, with 200 seeds per experiment and confidence intervals.

Weaknesses

Metric definition: - The effective hyperparameter dimensionality depends on the total number of hyperparameters of an algorithm and is likely to scale with it, so this makes it incomparable between algorithms with a different number of hyperparameters. Perhaps counting the number of parameters that changed instead would make it more comparable across algorithms, as is what would dictate the budget of practitioners eventually. Hyperparameters: - I would consider the minibatch size and the number of epochs in PPO to be critical hyperparameters as well. It's not clear how the choice of hyperparameters to sweep over was made. - (minor) The epsilon in the denominator of the minibatch normalization may also play a big role. Inconsistent/confusion notation: in line 91, the tuple (w,h) defines an agent (a,), but this is used in a confusing and inconsistent way throughout the paper. - line 116 $\hat p(a, e, h): if $a$ is there then $h$ is redundant. It seems like $a$ there stands for $w$. - Equation 1 same. The $a$ should be a $w$. - Equation 2 and 5: 2 and 5: $\Gamma(a, e, w)$ not it's even more confusing there is both an $a$ and an $w$ and there is an $h$ missing. Claims: Line 321 "vastly difference effective hyperparameter dimensionalities": the largest gap observed in the paper is from 2 to 4, so I would not extrapolate here, though arguably that's indeed an additional 2 dimensions to sweep over so scales exponentially.

Questions

Reporting the mean under the curve (AUC) with 95% bootstrapped confidence intervals over 200 runs is great, but I would expect some discussion on the use of a confidence interval, as it collapses the shaded area across the 200 runs to the statistic being computed (here the mean AUC). To me, using a dispersion measure like the variance of the mean AUC would also be valid, as some hyperparameters would have more variance than others (although this could also be seen as an additional dimension of sensitivity). I would appreciate it if the authors could comment on this. Also, at what point are the confidence intervals computed? When computing the expected performance or when computing the hyperparameter sensitivity, etc?

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors adequately state the limitations of their work and its broader impacts. In particular the potential limitation of the framework to a specific environment distribution.

Reviewer ydma2024-08-08

A promising paper that needs more polish

I thank and commend the authors for the strong points laid out in the global response, the revision of their figures, and their direct comments. I find figure 2 in the additional pdf (global response) extremely interesting and am surprised by the extreme contrast in results when comparing figure 4 (main paper) with figure 1 (pdf rebuttal). --- I agree with the point that mujoco/ brax was only a test-set to showcase your visualization method, and I did not find any overly strong claims or conclusions in these sections in particular. However, the main problem that I have is that despite the costs of the experiments, and the proposed visualization, I can still not draw any insights from these results. Which is unfortunate. The issue that I had is that this section has too strong of a focus on the results, for example, section 4.2 is a lengthy discussion of most parameters (the details of which, I believe were a bit beside the point) and section 4.3 simply enumerates everything we can see in figure 4. I want to see a critical discussion of: how can we use this visualization method to gather insight and what can we do with this information. This should be accompanied by a well executed test-case. This should include multiple ways to look at the same data (different normalization, environment subsets, conditional parameters, or more). Like I mentioned before, I think figure 2 of the rebuttal-pdf adds tremendously to this discussion. --- Considering all my previous points, I still think this paper lacks polish. It looks promising, but it is not quite there yet. I will raise my score from 3 to 4 and contribution from 1 to 3, since the authors did address many concerns.

Authorsrebuttal2024-08-13

Here are two insights that can be drawn from Figure 2 of the pdf rebuttal. While it has been known in the literature that observation normalization matters a lot for performance, we can now see that the performance gain comes with, or is even partially enabled by, increased sensitivity. It has also been previously reported that advantage normalization does not matter for performance. This appears to be true. However, it somewhat lowers sensitivity while retaining similar performance, making it possibly more interesting for practitioners than the literature has presented it. Thank you for pointing out that section 4.3 could be deeper. For the camera-ready, we will drastically shorten section 4.2 (or move it to the appendix) and expand section 4.3 to include additional discussion and figures (different normalization schemes, environment subsets, etc.) as discussed in the rebuttal and responses.

Reviewer yuTN2024-08-08

Thank you for extensive response to my review. A few thoughts are below. Where satisfied, I have not included response to each point for brevity's sake. # 2D plot I think the key for me about the colouring is that it enforces a fairly arbitrary segmentation that I think gets in the way of analysis. I think, personally, I would prefer to see results simply plotted on these axes, with analysis separated into text. # Comparison with AutoRL While I agree there is a difference, I think the takeaway is that both AutoRL and your work have similar motivations - to promote systems where an environment is put in and a policy comes out without human input. As such, I still think this is worthwhile comparison. In a sense, you can consider that AutoRL algorithms are 'hyperparameter free' in that they deal with hyperparameters internally, and thus have no hyperparameter sensitivity. --- In addition to the above, I still feel there is a missing component considering how human-in-the-loop and prior work would focus a lot of these search efforts by offering intuition about the kind of hyperparameter ranges which are useful. That said, I am so far satisfied that some of my concerns have been addressed sufficiently, and thus have increased my score from 3 to 4. I remain open to discussion about the above.

Authorsrebuttal2024-08-13

2D Plot While color may not be the best choice for visualizing these segmentations, we don’t believe the segmentation is arbitrary. Each of the segmentations has a different interpretation of its relation to the reference point. The slope one line passing through the reference has unique importance as it marks the points where observed performance gains are directly attributable to per-environment hyperparameter tuning. Comparison with AutoRL We agree that, like AutoRL, we are interested in promoting methods that require less human intervention and tuning to apply. The key difference we see between our work and AutoRL is that we are demonstrating the utility of our methodology with an experiment on PPO, not proposing a new way to tune hyperparameters in RL. While AutoRL methods tune an RL algorithm's hyperparameters internally, it should be noted that the AutoRL algorithms themselves often have hyperparameters (e.g., the scaling factor and crossover factor parameters in DEHB). For the camera-ready, we will include an additional discussion of the AutoRL literature and how the AutoRL community could use our method to measure their effectiveness at improving performance while reducing (hyper-)hyperparameter sensitivity. Thank you for your additional comments. We appreciate your feedback and discussion.

Reviewer 6Bue2024-08-11

I thank the authors for the clarifications. Regarding the bootstrapped confidence intervals, what I mean is that the more samples available and the more points drawn at a time the smaller the interval will be. This is different from a dispersion measure, like the variance, which would not shrink with more data. I maintain my score.

Authorsrebuttal2024-08-13

Thank you for your response and for clarifying. We are sorry that we missed this point in our initial response. We used a confidence interval as we are interested in understanding the exact sensitivity and performance under different algorithmic variations and so plot the confidence in our sample estimates. We will include a table in the appendix that reports the standard deviation across seeds of the AUC for each hyperparameter setting tested in each environment.

Reviewer aSZK2024-08-12

answer to rebuttal

Thank you for your answer. 1. Thanks for adding this figure, I think it is important to verify this stability to make sure the statement of the papers are applicable in generalized settings. 2. Great to hear, as mentioned in my review, this data can be valuable for other research. 4. The analysis you suggest makes sense and would provide value (I gave f-anova as an example but any analysis on hp importance would be valuable). I agree with you that studying hyperparameter importance is tangential to the main point of the paper but still seems related (it may be that some algorithms have the same hyperparameters to tune and other requires a larger set which is also interesting for practitioners, in addition to know which hyperparameters had the most effect) 5. I get your point about CDF, I dont mind if you use (p, 1-p) percentile transformation instead as long as you make sure that the results are not completely tied to one normalization. Regarding the downsides of the CDF you mention, I think those are standards points not necessarily tied to your use-case: the CDF gets rid of the scale (only the ordering matter), this has benefits (robustness to outliers, uniform distribution obtained) and downsides (sometimes the scale is important) which is why this normalization is good to have in addition to min-max as it has different trade-offs. I have raised my score given that some of my points were addressed.

Authorsrebuttal2024-08-13

Thank you for your review and suggestions for improvement! We will incorporate them by modifying our broader impacts statement to include a discussion of the computational resources used as well as ideas for future work that promotes sustainable research.

Program Chairsdecision2024-09-25

Decision

Accept (poster)

© 2026 NYSGPT2525 LLC