Does TD3+ODT actually address the same support/coverage problem identified for ODT, or only add a heuristic local regularizer?
Great work. I have been interested in Decision Transformers as an alternative to offline RL, and I may be misunderstanding some parts since I have only read original Decision Transformers & Online Decision Transformers papers, so I would appreciate clarification or any misunderstandings due to lack o background in this domain
The paper’s motivation is that ODT can fail when the requested RTG is outside the return support of the offline/online replay data. I agree this is a plausible failure mode. However, I am not convinced that adding TD3 gradients theoretically addresses this issue, rather than adding a useful local actor-critic regularizer.
The paper argues that high-target-RTG conditioning depends on quantities such as
$$
p_\beta(G=g\mid s,a),\quad p_\beta(G=g\mid s),
$$
which are poorly estimated when (g) is outside data support. But the proposed TD3 actor term introduces
$$
\nabla_a Q_\phi(s,a)\big|*{a=\mu*\theta(z)},
$$
which is also reliable only in state-action regions covered by the replay buffer or subsequent online exploration. If the critic is queried at unsupported actions, then (Q_\phi(s,a)), and especially its action gradient, are function-approximation extrapolations. This seems like the standard offline/off-policy critic extrapolation problem.
Could the authors clarify what assumption makes the TD3 gradient reliable precisely where ODT lacks coverage? For example, is there any bound such as
$$
|\nabla_a Q_\phi(s,a)-\nabla_a Q^\pi(s,a)|\le \epsilon
$$
on the actor’s queried state-action distribution, or any support/concentrability condition such as
$$
d^{\pi_\theta}(s,a)\le C d_{\mathcal B}(s,a)?
$$
Without such a condition, it seems the TD3 term may inherit a similar support problem rather than solve it.
A related question is whether the method is theoretically doing more than TD3+BC with a DT actor. The actor loss is essentially
$$
|\mu_\theta(z_t)-a_t|^2
-----------------------
\lambda Q_\phi(s_t,\mu_\theta(z_t)).
$$
Locally, for small (\lambda), this suggests
$$
\mu_\theta(z_t)
\approx
a_t+
\frac{\lambda}{2}\nabla_a Q_\phi(s_t,a_t),
$$
i.e., a behavior-cloning action plus a critic-gradient nudge. This is reasonable, but it looks like a TD3+BC-style regularized actor-critic objective, with the main difference being that the actor is a Decision Transformer. What is the DT-specific theoretical contribution beyond this architectural substitution?
I also found the AWAC/Laplace derivation stronger than justified. The paper assumes both
$$
p_\beta(G\mid s,a)=\mathrm{Laplace}(Q^\beta(s,a),\sigma)
$$
and
$$
p_\beta(G\mid s)=\mathrm{Laplace}(V^\beta(s),\sigma).
$$
But generally,
$$
p_\beta(G\mid s)=\int \beta(a\mid s)p_\beta(G\mid s,a),da.
$$
Thus, if action-conditioned return distributions are Laplace with different centers, the marginal is generally a mixture of Laplace distributions, not a single Laplace centered at (V^\beta(s)). Is this lemma intended only as intuition, or does any main claim rely on it?
Finally, could the reported gains be explained by benign local coverage in the selected benchmarks? The method may work when the replay buffer already contains enough local state-action variation for TD3 to learn useful gradients, or when online exploration quickly expands coverage. A useful diagnostic would be an environment where the offline data contains poor actions with no local gradient information toward the better action, or where the optimal action lies in an unsupported action region. Does TD3+ODT still improve there?
Overall, I think the empirical heuristic may be useful, but the theoretical framing appears overstated. The paper convincingly argues that ODT can fail due to return-coverage issues, but I do not see a corresponding argument that TD3 gradients overcome those issues rather than adding a local actor-critic update valid only under usual replay-support assumptions.