Summary
This paper presents an approach to generation using diffusion models augmented with a reward function. It does so by setting up a semi-supervised learning setup, where the reward function is learned from a small set of data. The reward is then used to learn a reward conditioned score function, which is subsequently used to generate data conditioned on requested reward.
Assuming a linear subspace, the analysis approaches this setting through the lens of linear bandits, and characterizes the error or suboptimality in terms of regret to the target or requested reward, off-distribution error, and on-distribution error.
Experiments evaluate the above interplay, and how requesting higher rewards leads to distribution shift in the generated samples. Further experiments show how pre-trained models can be adapted to generate reward-conditioned samples.
Strengths
* Presents a practical way to add subjective rewards to generate data beyond concrete prompts like text
* Analyzes this reward conditioned generation setting and identifies the interplay between rewards and distribution shift.
* Experimentally verifies the claims made in the analysis.
Weaknesses
* The paper does not discuss a practical manner in to identify when the generative model starts deviating from the training distribution.
* Comparing the diffusion model's error to a linear bandit setting is interesting, but the exact connection between the two seems a bit murky. The training process does not seem to take advantage of any bandit learning algorithms.
* While it shows that the model can be adjusted to arbitrary rewards, it does not showcase a practical use case.
* The technical novelty is unclear.
=============================
### Post-Rebuttal
* The author's rebuttal and other reviews have made the technical contribution of the paper abundantly clear.
* Additionally, the connection to _offline_ bandits is more evident and does add value.
* The additional RL experiment grounds the claims made in the paper more concretely than the previous experiments.
As such, the majority of my concerns have been allayed with the author response.
Questions
* Section 2.1 gives the scale of $\sigma$ but does not mention any such constraint about $f^*$. Section 4.2 constructs a random reward, but this setting also makes no mention of the scale of the reward. How does the scale of the reward affect learning?
* If someone were to try and reproduce these results, how would they go about it? Perhaps an experiment that specifically tries to maximize some specific property that is hard to communicate through a text prompt could be shown here, to communicate the effectiveness of this approach, as well as to assist reproducibility.
* The labeled data is based on the CIFAR-10 dataset. Is the unlabeled dataset from a corresponding distribution? Does it have similar resolution, size, and kinds of pictures?
* The paper states `To optimally choose a target value, we must trade off between the two counteractive effects.` Are there any practical methods to do so?
* Reinforcement learning faces the problem of reward design. The analysis done in this paper could be useful for this problem of reward design. It would be useful to deepen the discussion on reward design and reference some related work [1, 2]
### References
[1] Booth, S., Knox, W.B., Shah, J., Niekum, S., Stone, P. and Allievi, A., 2023, June. The perils of trial-and-error reward design: misdesign through overfitting and invalid task specifications. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 37, No. 5, pp. 5920-5929).
[2] Knox, W.B., Allievi, A., Banzhaf, H., Schmitt, F. and Stone, P., 2023. Reward (mis) design for autonomous driving. Artificial Intelligence, 316, p.103829.
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Limitations
while the proposed approach opens up avenues to communicate and generate data using feedback other than text prompts, it does not sufficiently address problems that might arise from such freeform feedback.