Thank you for your review and for acknowledging the innovation and thought-provoking aspect. Hopefully we can address your concerns, as there seem to be some misconceptions about the paper.
1. There seems to be a misconception of the main theorem, as we make no dependency on knowing $Q^*$. This is in fact a primary motivation for the theory as described in the 3rd paragraph of the introduction: we want to “score match” to something unknown, so we need to develop new theory that doesn’t rely on knowing anything about $\pi^*$. The main theorem is this, as seen through its contrapositive (and more clearly in its proof): if the score $\Psi$ and action gradient of the Q function with respect to $\Psi$, $\nabla_a Q^\Psi$, are misaligned, then any amount of aligning $\Psi$ to $\nabla_a Q^\psi$ will strictly increase the Q-function towards $Q^*$, and is thus a valid update rule. A large part of this paper's novelty comes from the new math developed to prove this theorem in the appendix, which makes the theoretical role of the QSM algorithm much more clear. We understand the appendix is not required reading, so we have updated the main text to increase clarity.
2. Note in eq. (6) that the state and action are time-separated systems: for every state step, there are many action steps. For any Langevin dynamics-based method (including diffusion models) to work, there need to be enough steps so that the initialization at each state time step makes little difference. While the state-time initialization $a_{t-1}$ is less important, the internal diffusion steps using $\Psi$ are important, and our theory is applied to develop the QSM algorithm to update $\Psi$ based on these internal steps. *In summary:* initializing between $a_{t-1}$ and Gaussian noise makes little difference, but the latter has some practical benefits, such as the computational graph not connecting through time, and allowing for bang-bang control, etc.
3. The main practical difference: all previous works apply diffusion model policies to offline RL problems, while our work focuses on online/off-policy RL. While both try to maximize Q in some way, these are fundamentally different approaches under the hood. For a diffusion model of depth $K$ and denoising model $\phi$, Diffusion-QL needs to backpropagate through $\phi$ composed on itself $K$ times, resembling gradient structures and issues with RNN’s (vanishing/exploding gradients), while QSM only needs to differentiate with respect to $\phi$ itself (for arbitrary depth $K$) – using our theory to utilize the “inner optimization” of diffusion models allows methods like QSM by design to, unlike Diffusion-QL, train well with deeper diffusion models and improve as $K$ is increased, even during training or evaluation. QSM has implicit entropy regularization as described through the pedagogical example in 4.3, while Diffusion-QL needs explicit regularizers. These benefits come from our presented methodology, and the example algorithm QSM, being derived from and respecting the underlying score-based structure of diffusion models, which the direct Q optimization objective does not capture.
4. The emphasis of this paper is a new theoretical framework to update diffusion model policies, as well as a standard initial implementation in an off-policy RL setting to demonstrate that the theory works as intended and shows its promise, namely when there are less training samples. Much of this rich theory is developed and presented in the proofs in the appendix; while important for reading, they are quite long and dense, and would greatly hinder readability if crammed in the main body.
There are plenty of interesting directions to improve on QSM into a state-of-the-art continuous RL algorithm, and many other interesting ways to implement a Q-score matching algorithm (e.g. using $\nabla_a Q$ directly to sample via Langevin dynamics), but we feel the theory is dense and rich enough to deserve its own paper, and position it to lay the foundation for how to use diffusion model structure to design more efficient diffusion policy training algorithms and spark future research. If there are particular insights you hope to gain from Ant and Humanoid, please let us know.
*Questions*: Note $\pi_\phi$ is not modeled explicitly in diffusion models, but we do give how actions are sampled in the first line within the loop. The other questions should be addressed from the above response.
Hopefully this has clarified some points in the paper and addressed your concerns. If so, please consider modifying your scores and rating to reflect this. If not, we are of course happy to continue the discussion.