Summary
This paper introduces a novel approach to handling distribution shift in
off-policy RL by the means of Q-function regularization. This is
accomplished by modulating a penalty term that is overly conservative in
CQL. The authors argue that CQL overcompensates for the distribution
shift in cases where state-action pairs exist with non-negligible
density in the dataset, but are still less likely under the reference
policy than the learned policy. Based on this insight, the authors
propose to modulate the CQL value function penalty by a decreasing term
in the density of state-action pairs under the reference policy, which
activates as soon as this density crosses a threshold. Moreover, the
authors propose a prioritized sampling method to further reduce the
underestimation bias. The resulting algorithm, called EPQ, achieves
superior performance relative to its competitors across many familiar
offline RL benchmarks.
Strengths
The motivation for this work was fairly strong, and the authors identified an
interesting shortcoming with the overestimation correction in CQL. Figure 1
followed by Figure 4 do a nice job of depicting the influence of this
shortcoming, and how the proposed method corrects it.
Moreover, the EPQ algorithm is deployed on a large suite of benchmarks, and
outperforms all competitors with remarkable consistency. To complement these
results, the authors conducted experiments to verify that EPQ does in fact
reduce value estimation bias, shown by comparing EPQ value estimates with
Monte-Carlo estimates, as well as predictions from CQL. The experiments are
conducted over four random seeds. While this is relatively few seeds, I think
this is acceptable given the range of tasks that were tested.
Weaknesses
The presentation of the paper (e.g., writing, figures) can be improved.
In particular, many of the figures were difficult to read and/or
interpret. Both facets of Figure 2 took substantial effort to understand
for me (in fact, I think I would have more easily understood the paper
without having seen these figures; see Questions below).
The statement of Theorem 3.1 is not precise enough, particularly for the
latter claim. Moreover, I suspect there are some technical assumptions
missing; see Questions below.
Furthermore, I do not entirely understand the motivation for the
prioritized dataset. Particularly, it is not clear to me that Theorem
3.1 (the theoretical justification for EPQ) actually applies with the
prioritized dataset, since the prioritzation depends on the estimated
Q-function being updated. Beyond that, the ablation of this feature is
not very convincing.
While there is a wealth of empirical results, confidence intervals from
the baselines is largely lacking—this is especially relevant in Table 1.
See for example the `door-cloned` row: EPQ is identified as the best,
but its confidence region definitely overlaps that of CQL, and probably
those of many of the baselines as well. The same goes for
`relocate-cloned`. as well as (I'd suspect) many of the AntMaze tasks.
That said, the results do suggest that EPQ frequently outperforms its
competitors, and rarely does substantially worse.
Finally, it would have been nice to see stronger heuristics for choosing
the threshold parameter $\tau$. Figure 2a suggests that $\tau$ should be
a function of the amount of data in the dataset, but this is not
actually discussed anywhere. Rather, the authors claim to have found a
choice for $\tau$ that is inversely proportional to the volume of the
action space, but results for this choice are only given on one
environment, precluding any conclusion that this choice/trend is good in
general. The proof of Theorem 3.1 also gives a condition for determining
when $\alpha$ is large enough, which is roughly inversely proportional
to the lowest density under the reference policy over all states and
actions – therefore, choosing $\alpha$ to be inversely proportional to
the volume of the action space is only theoretically justified when the
reference policy is uniform.
## Minor Issues
The notation/definition of the Bellman operator $\mathcal{B}^\pi$ s not
exactly correct. In its definition on line 61, $\mathcal{B}^\pi$
averages over all next states $s'$. Then, in the expression on line 63,
you are taking an expectation over all state transitions $(s, a, s')$ in
the dataset with $\mathcal{B}^\pi$ evaluated in this expectation. Since
$s'$ isn't used anywhere in this expression explicitly, my assumption is
that you're using this state as the state to bootstrap from in the
application of $\mathcal{B}^\pi$; but then $\mathcal{B}^\pi$ on line 63
is not the same as its definition on line 61.
On line 83, you refer to the "actual average return $G_0$", but $G_0$
was defined as the random return (not averaged) on line 56.
Formatting of equation (2) is not nice – it almost looks like it's
depicting two separate formulas. It may read easier if you instead
colored the two factors and described their influence in the text below.
In Figure 1 and Figure 4, it would be very helpful to see where $0$ lies
on the y-axis on the estimation bias side.
Figure 2a is very busy and difficult to interpret. Firstly, I think it
would be better to highligh the magnitude of the penalty itself, as
opposed ot the penalty reduction (which implicitly depends on some
initial penalty, I'm guessing from CQL). Moreover, the relationship
between the amount of data and $\tau$ should be discussed before this
figure, even if superficially (e.g., with a sentence that says that
$\tau$ decreases as you collect more data). Then, the figure would have
a much more clear interpretation: increase the amount of data, and the
penalty will be relaxed more aggressively.
There appears to be a formatting error on line 140, "Proof) Proof…".
There is a formatting error in equation (5), $Q_(s, a)$ -\> $Q(s, a)$.
On line 243, $200k$ should be $200\mathrm{k}$.
In Table 1, the "total" rows are not good indications of performance,
firstly because the returns for the different environments are not
normalized. That said, the results for EPQ still look good if you
neglect the "total" rows.
In Figure 5, it would help to use a different line style to emphasize
which curve corresponds to EPQ. I found it difficult to distinguish EPQ
from CQL ($\alpha=0.0$) – fortunately these curves generally occupied
disjoint regions in the graphs.
Questions
In Figure 1, what is the relationship between $\tau_i$ and $N_i$
($i=1,2$)? Such a relationship has not been discussed up to this point.
In the proof of Theorem 3.1, I believe some assumptions are missing. If
$\pi(\cdot\mid
s)$ is ever supported on an action $a'$ such that
$\hat{\beta}(a'\mid s) = 0$, then $\Delta^\pi_{EPQ}\to\infty$.
Therefore, there would be no $\alpha$ large enough to underestimate the
value function for your argument on line 465 as long as there exists a
single $(s, a)$ in the dataset for which $\xi^\delta(s, a) > 0$. Since
$\hat{\beta}$ was defined to be the empirical conditional distribution
over actions from the dataset, this result actually suggests that
$\alpha$ must be infinite whenever your dataset does not fully cover the
action space (which is always the case in the experiments, where the
action space is continuous).
In table 1, why aren't confidence intervals given for the baseline
methods?
Why are there no confidence regions shown in Figure 6? Particularly, it
would have been helpful to see these in Figure 6a. As it stands, the
effect of the prioritized dataset depicted in this figure is a little
underwhelming.
With regard to the analysis of the penalty threshold, why should we
scale $\tau$ linearly with the density of the uniform distribution over
$\mathcal{A}$ (that is, inversely proportional to the volume of the
action space)? Figure 6b does not indicate whether the choice of
$\tau = 0.2\rho$ is actually a good choice across environments, so
indeed it could have just been that $\tau=0.2\rho$ happens to work well
in `hopper-medium` by chance. You have have investigated (or at least
presented the results) that show how this form of scaling with the
action space performs.