Summary
For the reinforcement learning (RL) setting, many problems are cast into a fixed discrete time sampling of the true underlying system. This paper investigates given some fixed data budget available, what is the optimal sampling rate in terms of temporal resolution to balance the trade-off between approximation and statistical estimation error. Theoretical results are obtained for the exact form of these errors is derived for the MSE of a Monte Carlo in a Langevin process for both finite and infinite horizon setting. This shows that there is an optimal choice in temporal resolution to balance these errors, and this is verified by simulated linear quadratic systems. Similar kind of trade-offs can be found in empirical (nonlinear) environments from gym and MuJoCo.
Strengths
This is a well-written paper which looks at a problem in value estimation that does not appear to have been explicitly studied on this level before. There is detailed theoretical analysis for the case of a Langevin process, and the results are interpreted in a practical way. The demonstration of these theoretical results in numerical simulations in Section 4.1 is very clear. Verification of similar effects on empirical environments is useful to see.
Overall, this paper contributes to the literature on additional considerations that should be made in continuous time RL setting, and it would be useful for future developments in this area.
Weaknesses
Practically the policy is varying at the same time, and perhaps the same step size should not be used over the course of training. Therefore it may be that the practical impact on implementation of this paper may be limited.
For Figure 3, it seems to be misleading to fit a line matching $h = \mathcal{O}(B^{-1/3})$ for the empirical experiments. Particularly given lines like for that of the InvertedDoublePendulum, clearly a different gradient should be considered. The observation on these empirical experiments are interesting, but there is no need to force the results on these nonlinear systems to match the analysis in the previous section.
In setting the step-size $h$, this relies on the ability to perform experiments on a smaller data budget first, which may not always be possible in practice.
Questions
How might an adaptive sampling procedure affect the results?
It is assumed that the estimation horizon is the same length as the episode, this seems like a strong assumption which does not hold in a lot of applications (i.e. you do not know how long the episode is a priori, nor are the episodes the same lengths). How would the results in finite horizon change in this case?
Should the $h$ on line 165 be $h^*$?
Can you clarify where the MSE_T between line 168 and 169 come from?
The second term on the right hand side for the MSE_T has denominator $hB$, will this assume $h = \mathcal{O}(B^{-1/3})$ therefore the second term is $\mathcal{O}(B^{-2/3})$ hence till tends to 0?
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Limitations
The theoretical results obtained seems to assume a fixed observation/estimation horizon T, that is pre-set, but since episode are not of fixed lengths in RL settings, and indeed may increase as the policy improves, this may affect the validity of the theoretical results.