Summary
This theory paper proposes an algorithmic framework where the "hypothesis" (model for MB methods, Q function for MF methods) for each iteration is chosen by maximizing one objective: the hypothesis likelihood (i.e., NLL for MB methods, TD error for MF methods) plus the expected returns (value function at the initial state). The paper provides regret guarantees matching prior work. The paper proposes a practical variant of the method (building on top of MBPO and TD3) that shows some promise on sparse versions of benchmark continuous control tasks (while matching performance on the standard, non-sparse versions).
Strengths
**Originality**: To the best of my knowledge the proposed frame work is original.
* One prior paper that's related to the model-based version of this paper is https://arxiv.org/abs/2110.02758. This that paper lacks the theoretical contributions of this paper, and comes at the problem from a different perspective, the resulting algorithms look surprisingly similar: both learn a model that is optimized to (1) have high likelihood and (2) produce high-return policies. I'd be curious to see if the current paper also performs well on the stochastic gridworlds used in the prior paper to showcase the benefits of the "optimistic" model.
**Clarity**: I generally found the writing clear, which is especially impressive for a large theory project packed into 9 pages. The bottom of this text block has a few suggestions for improving clarity. A few high-level things
* The related work section is generally well written, including a pretty thorough review of prior work. One suggestion is to explain the differences from prior work. What are the limitations that this paper will address?
* I didn't get much from reading Section 5; I'd prefer that the space be spent on more intuition for the previous section, or more empirical results.
**Significance**: Overall, it seems like the proposed framework is a novel and potentially useful way of designing better RL algorithms. I especially appreciate the simplicity of the proposed approach. I think that the paper may be a bit closer to prior work that the paper makes it out to be (see comment under Originality, and the comment about Gumbel), but it still seems to represent somewhat of a departure from the conventional ways of thinking about RL algorithms. Like most new algorithmic frameworks, it seems like there may be a few kinks that have to be worked out empirically (how to choose $\eta$; is there a way of implementing the model-free version without the extra CQL term). Nonetheless, it seems like the paper could inspire more work in this direction.
**Strengths**
* The paper is generally well written, includes a thorough review of much prior work.
* The proposed framework is derived for both model-free and model-based algorithm.
* An empirical version of the proposed method is applied to to reasonable continuous control tasks. This is really nice to see in a theory paper.
Weaknesses
* In a few spots, I felt like the paper claims were not entirely substantiated:
* "One Objective to Rule Them All" in the title -- Without comparisons to "all" prior methods, I don't think this is a fair claim. Perhaps a more accurate title would be "An Objective Fusing Estimation and Planning for Exploration"
* "algorithms predominantly undertake three tasks" -- This seems to be describing methods based on Thompson sampling and posterior sampling, but I'm not sure that R-MAX-style methods fit this mold
* "outperforms MBPO by a stable margin ... showcases greater sample efficiency" -- I'm a bit unsure what these statements are referring to. My read of Figure 1 is that there might be statistically significant gains on 2 / 8 tasks (cheetah-vel-sparse, ant-vel-sparse). I would still vote for accepting the paper even if it only outperforms baselines on 2/8 tasks (but I think that accurately portraying the results is very important); this is especially true if we see large benefits on some tabular settings (see suggestion under Originality).
* The paper alludes to "data-dependent level-sets" in a number of places, but never defines what these are. I'd recommend either including a few sentences describing similarities/differences, or moving this discussion to the appendix.
* I have a few questions about some of the technical details (see below).
* From a practical perspective, the model-free version of the method seems to violate conventional wisdom: many practical TD methods (e.g., TD3, MBPO) choose the Q function that has lowest Q-values, whereas the proposed method chooses the one with highest Q values. Not only does this make be a bit concerned that the theoretical framework might not always work well in practice (as evinced by the need for the additional CQL regularizer), but it also makes me wonder exactly how the model-free version of the method is implemented when combined with TD3 (does it take the min or max of the Q functions)? At least for the model-free version, one thing that might help would be to sure that using the CQL regularizer allows the _same_ method to achieve good results in both online and offline settings.
---------------------
**Minor comments**:
* In the abstract, mention that this is going to be a theory paper. I expected this to be an empirical paper until partway through the intro, when I realized that all of the citations were referring to other theory papers. As such, I found the claim that prior methods "involve impractical algorithmic components" a bit strange, as prior _practical_ methods cannot require impractical components.
* L45 -- L47: I found this part a bit unclear because it was unclear what the optimization variable is (a policy, a model, data). This point is clarified later.
* L50: I found this part a bit unclear because it was unclear what the hypothesis class was over (policies, models, etc).
* Some small grammar errors throughout. An automated grammar checker should catch these (e.g., dropped articles)
* L80 -- This definition could be cut, as it's already included a few paragraphs earlier. Cutting this would make it possible to remove some of the whitespace hacks.
* L101 -- What is "BE"?
* Empirically, in many TD algorithms the TD error _increases_ as the policy improves. The standard explanation is that collecting non-zero rewards results in higher TD errors. This seems potentially related to the analysis in this paper.
* L260 -- Including a table for comparing these prior methods might help.
Questions
1. How important is the value of $\eta$ for both the theoretical and empirical results?
2. L147 "Without loss of generality" -- Can the authors elaborate on this point? My guess is that we can ignore stochastic rewards because we could define a new MDP that has a copy of each state for each possible reward that that state could have (and update the transition probabilities accordingly). I'm not sure why it's OK to assume that the reward function is known.
3. There's a sense in which the proposed method resembles posterior sampling methods, by drawing a connection with Gumbel-softmax-style sampling. One way to sample a random variable is to _deterministically_ choose the value that maximizes likelihood + noise, for an appropriately chosen noise distribution (e.g., the Gumbel distribution for categorical distributions). I'm curious if we can interpret posterior sampling methods as a version of Eq. 3.1 where the first term ($V(x)$) is replaced by an appropriate noise distribution (which has nothing to do with the value).
4. L206: Does $V_f^\pi$ make sense for both the model-based and model-free versions of the method? For the model-based version, I'm interpreting this as the expected returns of policy $\pi$ under model $f$. But for the model-free version, I'm unsure if this is supposed to be the expected returns of $\pi$, or the estimate from (value function) $f$.
5. L206, "reduced to vanilla actor-critic": Is this true for the model-based version of the method?
6. Is the second term in Eq. 3.3 required?
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Limitations
Limitations are not discussed in the main paper. The appendix has a cursory paragraph on limitations. I would recommend including the limitations in the main text, and including a transparent discussion of benefits/limitations of the proposed method relative to prior methods. For example, the first term in Eq. 3.1 seems to sometimes cause optimization problems. Ensemble-based methods might be more stable.