Summary
The authors study how batch size affects RL performance, and argue that a reduced batch size might (quite surprisingly) bring better performance improvement in a number of settings, in particular for QR-DQN, a smaller batch can lead to much better performance (almost doubling the performance). Different batch sizes are tested in Atari environments together with other hyperparameter changes. The authors also point out a benefit of using a smaller batch size is the reduction in wall clock time. The paper focus on empirical study and analysis, and provide a list of interesting findings.
Strengths
**originality**
- the paper dedicates to bring a better understanding of effect of smaller batch sizes with extensive empirical studies, although there are already past works that study the effect of batch sizes, the results in this paper bring some new findings and observations and can be considered novel contribution.
**quality**
- overall presentation is good, but some arguments can be improved.
- extensive experiments and ablations are great
**clarity**
- Overall paper is clear to read. And the structure is easy to follow.
**significance**
- the observations presented in the paper can be interesting to the research community and help us better understand the effect of different batch size settings
- the authors argue that a smaller batch size can bring wall-clock time reduction and given the results they also have potential to bring better performance, which is a result that can be helpful towards better algorithms
Weaknesses
Major concerns:
**related work**
- the authors claim "Surprisingly, to the best of our knowledge there have been no studies exploring the impact of the choice of batch size in deep RL." well, there are indeed some works that touch on this issue, for example to list a few:
- Accelerated Methods for Deep Reinforcement Learning by Adam Stooke and Pieter Abbeel.
- Reproducibility of Benchmarked Deep Reinforcement Learning Tasks for Continuous Control by Islam et al.
- Shallow updates for deep reinforcement learning by Levine et al.
- An Empirical Model of Large-Batch Training by McCandlish et al.
- these are older papers, please do some search on google scholar and have a better discussion of related works. Some of these works found larger batch size to be more beneficial. The authors need to spend a bit more effort in looking at related work and try to explain the discrepancy.
**Arguments and conclusions made in the paper**
- Some of the arguments can be improved, for example line 95: "In Figure 3 we can observe that, in general, reduced batch size results in improved performance." I don't think this is true, in Figure 3 there are 8 curves in 4 figures that have lower batch sizes than default, and 4 out of these 8 curves have weaker performance than the default, while the other 4 one can argue they are stronger or slightly stronger than default. One can argue in QR-DQN batch size of 8 is really good, but from this figure alone I don't see how it's a general trend.
- Line 226 Figure 11 (third column), I am not convinced there is a clear correlation between batch size and gradient norm. The variance is high, and for Asteroids and SpaceInvaders it seems in late stage of training they start to get higher gradient norms.
- Line 242 authors argue that " it is possible that the network is better able to adapt to an earlier rank collapse than to a later one." I am not sure how this argument is made, figure 11 col 5 shows that Srank does not positively correlate with performance at all. It seems this collapse has no negative effect on the training or even indicates stronger performance, which is again kind of going against what has been argued in previous literature. And the argument that "Smaller batch sizes seem to result in networks that are both more expressive and with greater plasticity." seems to be entirely wrong to me, as figure 11 shows on SpaceInvaders, small batch size of 8 has lowest srank and highest percentage of dormant neurons.
- Overall, I found a number of the points made in paper to be only supported by very weak evidence and they are not convincing. The authors might want to either modify the arguments into more accurate ones, or try to find better evidence to support the arguments. If the evidence is weak, the conclusion might not hold at all and it could be really coming from randomness or because of excessive fine-tuning on a particular algorithm on some particular environments.
The paper is currently lacking in these 2 aspects, but can be a good paper if these concerns are properly addressed.
Minor concerns:
- **originality** the novelty of the work is reduced by the fact that it is focused on studying existing methods, but mitigated by the novel empirical results, ablations and analysis.
Questions
Suggestions as discussed in weakness section:
- spend more effort on related work and explain discrepancies from conclusions in the previous literature. After looking at these previous findings, why do you think your results are different from these findings that mostly say larger batch sizes are better?
- go through the arguments made in the paper, modify them to be more accurate or bring in better evidence to support them.
Rating
4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.