Summary
This paper proposes an approach to censor the sample generation of pre-trained diffusion probabilistic models to better align with human preferences. The authors use minimal human feedback (<3min spent in providing the feedback for basic tasks, and <15min for more complicated tasks that they consider) to train a light-weight reward model, then use the trained reward model to provide either time-dependent or time-independent guidance for the generation process. The authors also use techniques such as ensemble, iterative imitation learning, transfer learning, backward guidance and recurrence to improve censoring performance. Experiments on MNIST, LSU church, ImageNet, and LSUN bedroom showed the effectiveness of their proposed method.
Strengths
The paper studies an interesting and important problem of aligning diffusion models with human preferences. By training a relatively lightweight reward model to guide the generation process instead of fine-tuning the large pre-trained model, their proposed approach could potentially save a lot of computation and human feedback.
Weaknesses
- The paper's structure could be more coherent and consistent. The authors introduce classifier guidance in Section 1.1 but leave its relation to censored sampling unclear until Section 4. This disconnect could confuse readers. Similarly, there is a lack of clarity regarding the reward function $r$ and the reward model $r_\phi$ introduced in Section 2. While initially, it seems that $r_\phi \approx r$, the possibility of $r_\phi$ becoming time-dependent, as mentioned in Line 97 of Section 3, introduces confusion because $r$ is time-independent. This is not resolved until Section 4, when time-dependent guidance is addressed. To improve readability, it would be beneficial for the authors to introduce the underlying notations and mathematical concepts of the entire model—from the diffusion probabilistic models to the reward models and their role in censored sampling—before discussing training and evaluations.
- In section 3.1, the authors choose $r_\phi$ as the product of $K=5$ independently trained reward models. However, they did not have enough discussions or experiments to back up these choices, which makes the resulting model seem arbitrary. In particular, since the authors mentioned $r_\phi(X)\approx r(X)=P(Y=1|X)$, it's unclear why taking the product--as opposed to taking the average--makes sense because each $r_{\psi_k}$ is also approximating $r$. Moreover, if we ignore the mathematical implications for now, the authors explained in line 115 that taking the product is essentially asking for unanimous approval, but readers might still question whether other methods, such as increasing the value of $w$ in the "union" method, could work just as well.
- The paper highlights extreme human feedback efficiency by saying that a few minutes of human feedback are sufficient. However, this claim overlooks the complexity of tasks considered in this paper, which are relatively simple compared to tasks where RLHF is typically employed in large-scale models. Therefore, it would be better to directly compare with previous approaches (such as fine-tuning) under the same task. Additionally, the more complex tasks, like censoring distorted bedrooms, actually require more than the estimated 3 minutes of human feedback. Hence, the title could be seen as potentially misleading or exaggerating the method's efficiency.
Questions
- For the training data $X^{(1)},\cdots,X^{(N)}$ in Algorithms 1 and 2, are they images corrupted by the VP SDE or images generated by the pre-trained model $\varepsilon_\theta$? What are the objectives in training each reward model in the ensemble method?
- In Algorithm 1, for each $k$ the learner randomly select with replacement $N_M$ benign samples. Does this step also need human feedback?
- Typo in line 204. "30 malign and 150 malign samples"
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.