Compositional Sculpting of Iterative Generative Processes

Summary

This paper proposes an approach of Compositional Sculpting for iterative generative models, including GFlowNets and defusion models. The model uses classifier guidance to sample from the target posterior distribution composed of pre-trained base models. The paper also proposes a training algorithm for the classifier. The approach is validated by empirical analyses of an image dataset and molecular generation.

Strengths

- The method is general enough for different GFlowNets and defusion models. - The empirical results validate the method.

Weaknesses

(1) It would be convincing to have more complicated experiments, especially for image data. Colored MNIST might be too small and simple. (2) More clarity might be helpful for the following points. (2A) What is the model in line 193? "Under the model we have introduced the variables y_1, ... y_n are dependent given a state s in S, but, are independent given a terminal state x in X." (2B) I am confused while reading line 209. The sampling scheme is 1) sample y from its prior 2) sample tau given y. But the following sentence says sampling y given tau.

Questions

Please refer to the weakness section. Typos - Line 217: train train

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

1: Your assessment is an educated guess. The submission is not in your area or the submission was difficult to understand. Math/other details were not carefully checked.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

The paper has a limitation section. It seems not to have a dedicated section for social impact.

Summary

The paper describes a way in which, given sequential samplers from multiple probability distributions, a combination of the samplers can be used to sample from a composition of the distributions. To be precise, the sequential samplers are either GFlowNets or diffusion models, the combination of samplers is a weighted combination of action distributions at each intermediate sampling step, and the composition of distributions can be defined by simple soft conjunction and set difference operators. This is demonstrated in toy illustratory experiments and multiobjective molecule synthesis (GFlowNet) and MNIST with digit class and colour attributes (diffusion).

Strengths

From the perspective of someone who works on both GFlowNets and diffusion models, this is a very well-written paper. - The text reads naturally and the right amount of detail is given in the main text. There is a good choice of illustrations to help the reader. - The composition of multiple GFlowNets has not been considered before and could be useful, especially in multiobjective problems. The unifying perspective on classifier guidance is also an advantage (but see below). - Code is provided, a nice addition to the paper. - I checked the GFlowNet-related math and believe it to be sound.

Weaknesses

- Line 234, 255, 634, 643, maybe others: typo "GFLowNet" - It would be good to explain why / state as a subclaim that (8) is a policy (i.e., sums to 1 over $s'$), which is not actually obvious from the definition. - It relies on the fact that $p(y|s) = \sum_{s'}p(y|s')p(s'|s)$, which follows from conditional independence of $y$ (a function of the final state) and $s$ given $s'$. That is a consequence of the Markov property in GFlowNets. A note should be made about this. - The equality may not actually hold in practice, when $p(y|-)$ is a trained classifier, so (8) may not exactly sum to 1. What do you do in this case (in the experiments)? - The results on molecule generation raise a few questions: - The reward exponent was set to $\beta=32$ or $\beta=96$, which is far larger than in past work, where it was at most 16. Why was such a choice made? This is suspicious, since convergence and mode collapse issues worsen at low temperatures. - Related, with such high exponents, one wonders about mode coverage in the learned distributions. Have you considered the in-sample diversity of the generated molecules (e.g., as measured by average Tanimoto similarity or diverse top-k metrics)? - On related work: - There is no substantial discussion of related work in the main text, even though there is a large body of work on compositional generation and classifier guidance with diffusion models (e.g., the many papers cited in the second paragraph of the introduction). - In the Appendix, the connection with [23] is discussed. The proposed method can easily turn a collection of classifiers for different objectives into a classifier for any convex combination of the objectives. It would be interesting to empirically compare this with conditioning of the model on the linear scalarization weights used in [23]. - The paper would be stronger if more explicit unifying connections were made between guidance in diffusion models and in GFlowNets. Note that diffusion models in a fixed time discretization are actually GFlowNets of a certain structure (cf. "A theory of continuous generative flow networks" [arXiv:2301.12594, ICML 2023] and "Unifying generative models with GFlowNets and beyond" [arXiv:2209.02606]). Classifier guidance using the gradient of $p_t(y|x_t)$ should be the continuous-time limit of equation (8).

Questions

Please see "weaknesses" above.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

4 excellent

Presentation

4 excellent

Contribution

3 good

Limitations

Yes

Summary

The paper studies the problem of composing independently trained generative processes of diffusion-based generative models and GFlowNets. The paper considers a setting where one has access to $m$ pre-trained samplers for $\{p_i(x)\}_{i=1}^m$, and the goal is to obtain a sampler which corresponds to a composition of these processes. Specifically, the authors consider two ways of composing the processes, namely harmonic mean: where the likelihood of the composition is high only where the component processes have high likelihood and contrast The authors frame this as sampling from a conditional distribution $p(x|y)$, where y is an observation which denotes the index of the process a sample is generated from. This results in a procedure analogous to classifier guidance which is popular in the diffusion literature. The authors show how the classifier guidance results in sampling from the desired composition. A critical component of this procedure is learning the classifier $p(y|s)$, for which the authors propose a MLE based procedure using trajectories from the base model. The authors first validate their approach on some synthetic tasks on a 2D grid, followed by experiments on small molecule generation with GFlowNets and colored MNIST with diffusion.

Strengths

* The paper studies an interesting question - which is relevant to the community. In particular, methods to leverage pre-trained models for various downstream tasks are becoming increasingly important with the growing adoption of pre-trained models for various domains. * Since similar classifier guidance approaches have been studied extensively in the literature on diffusion models, the novelty is relatively limited. Nonetheless, there are several technical aspects of the approach such as the classifier training scheme that are novel (to the best of my knowledge). * The proposed method is relatively simple conceptually, and in terms of implementation. * The experiments are well designed, and the results are quite promising, albeit with some caveats I mention below. * The paper overall is quite well written and easy to follow. I also appreciate the authors including the code with the submission.

Weaknesses

* As the authors discuss in Section 3, their theoretical analysis is analogous to classifier guidance in diffusion models. On the other hand, [1] establishes equivalence between GFlowNets and diffusion models. As a results, it seems to me that the insights provided by the theoretical analysis aren’t particularly novel even though the path to achieving them was different (which could be seen as useful on it’s own) * The experiments on diffusion are limited to a simple coloured MNIST task, with no baselines from the classifier-guided diffusion literature. * The central aspect of the approach is learning the classifiers, however there is no analysis on the classifiers - e.g. how accurate are the classifiers? what is the effect of the classifier performance on the results of the composition? How does the training of the classifier compare to simply training a GFlowNet from scratch in terms of runtime? * (Minor) A simple baseline that is missing in the experiments is training a GFlowNet with the appropriate composition from scratch. [1] Unifying Generative Models with GFlowNets and Beyond. Zhang et al. 2022. arXiv:2209.02606

Questions

* Could you please address the questions about the analysis of the effect of the classifier along with some runtime details? * What are the challenges in applying the method to more complicated tasks with diffusion models? * One natural question I had after reading the paper is that since the method encompasses GFlowNets and diffusion models, is it conceivable to be able to compose a mixture of GFlowNets and diffusion models on mixed continuous/discrete tasks? * In the experiments, the authors consider a maximum of 3 models being composed. Is there a reason to be limited to composing only 3 models? If not at least in the 2D example an experiment with more models might be helpful. Minor: * The main PDF on OpenReview appears to have some old version of the Appendix at the end. I assume that was by mistake? * In several places there is inconsistent usage of “GFLowNet” in place of “GFlowNet”.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

3 good

Contribution

2 fair

Limitations

The authors already discuss some limitations of the approach - effect of the classifier as well as the quality of the underlying models. I would also add limited evaluation and lack of comparison to existing approaches.

Summary

The paper proposes a method to compose multiple iterative generative models, i.e., either multiple GFlowNets or multiple diffusion models. The idea starts out with a mixture model over the generative models. Then, one can construct a categorical distribution over the generative models that tells us which model a sample originated from. By adapting classifier guidance to GFlowNets, the proposed method can compose multiple models in a way that allows both emphasizing or de-emphasizing specific models by treating the different generative models as different classes. On a diverse set of (toyish) experiments the method is shown to be effective for both GFlowNets and diffusion models.

Strengths

The method is very interesting. In particular, the part where the question "which model was this sampled from" is treated as a classification task for the purpose of compositional generation. The paper addresses a very important problem with high impact. The presentation is easy to follow and the text is well-written.

Weaknesses

The experiments clearly demonstrate the effectiveness and versatility of the proposed method. Though, the experiments are limited to toyish settings and I believe more complicated settings would greatly enhance the impact of this work.

Questions

All state-of-the-art diffusion models for images allow for text conditioning. Conditioning on different texts can be viewed as creating multiple different generative models from the viewpoint of this paper. I believe this is a more realistic setting than assuming that one would want to combine multiple diffusion models that were trained entirely separately from one another. It would be interesting to see some evaluations of this setting. A cheap strategy to obtain a classifier for this could be to fine-tune (part of) the CLIP model.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The main limitation is inherited from classifier guidance: the need to train a classifier on intermediate states. This is already mentioned in the paper.

Summary

The current paper focuses on the challenge of composition generation from pretrained generative models, with a specific focus on GFlowNets and Diffusion models. In comparison to prior literature, two novel compositionality operations are introduced for generating samples that are simultaneously likely according to two generative models or likely per a subset and unlikely per the remaining models. This is a strict generalization of operations introduced in prior work on composition of energy-based models. Practically, the operations are instantiated via a framework motivated by classifier guidance in diffusion models. Experiments are conducted on a molecule generation application and a colored MNIST problem. Rebuttals acknowledgment: I had a good view of the paper before rebuttals and the authors' response to my questions was fair. I continue to keep my score accordingly.

Strengths

I really like this work! The contributions are simple and straightforward, but very interesting. The formalization, generalized operations, and relation to classifier guidance were exciting to read through. The authors also appropriately acknowledge the limitations of their work, specifically the need for sufficiently strong component models.

Weaknesses

- My biggest apprehension is limited experimental investigation, which would raise the quality of this paper quite strongly in my opinion. I do not hold this to be a strong weakness though.

Questions

As noted in the introduction, use of large scale generative models has been seen in several tasks in the foundation model era of machine learning. Given the limited empirical evaluation, I would like to gather the authors' thoughts on use of their frameworks for, say, controllable generation of language via LLMs or controllable generation of images via text-diffusion models (e.g., what would the model show for known compositionality failures of Dall-e and related methods? See [1]). [1] Conwell and Ullman, 2022. https://arxiv.org/abs/2208.00005.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

4 excellent

Limitations

- A clear note of what is meant by compositionality would help in this work, since the term is extremely overloaded. The experiments currently reported focus on what would be called systematicity or systematic generalization in my opinion (see Hupkes et al., "Compositionality decomposed"), but other valuable forms of compositionality, e.g., productive generalization, will arguably require some "chaining" operator that allows prior generated states to be fed into the model for generating the next state. Since the work focuses on GFlowNets and diffusion models, where a notion of sequential generation of intermediate states is present, arguably authors can use their defined operations to perform productive generalization as well? I would appreciate if the scope of this paper is clearly discussed.

Summary

This paper introduces a method to combine sequential generative models, in this case GFlowNets, so as to create new distributions from base models. This is done by training classifiers that are then used to guide sampling. The method is tested on a simple grid and a molecular domain (emulating the problem of the paper that introduced GFlowNet), as well as on MNIST with diffusion.

Strengths

The paper is moderately easy to read, although some of its results were not immediately clear to me so I spent quite some time doodling on paper to convince myself that the propositions were reasonable. What the paper proposes and seems to be able to achieve empirically is very interesting. Combining generative models in the ways shown here could be an amazing multiplier of large pretrained models.

Weaknesses

Generally the paper is not making a good job of convincing the readers that the proposed method should work at the theory level, and that the effort of combining distributions by training a classifier is worth it (compared to retraining a generative model).

Questions

l211 and around, I’m not sure I understand the move to approximate $p(y_i|s)$ with $p(y_i|x)$, even when $s$ and $x$ are related by $\tau =(s_0,…,s,…x)$ a valid trajectory. How or when are these two quantities interchangeable? The objective in 12 involves $w_i$, but the paragraph following (11) seems to suggest that $Q(y_2, .., y_n|x)$ can be replaced by the $w_i$s. So I’m confused whether (12) uses the loss described in (11) or a modified version of it. Another worrying aspect of eq (12) is that there’s an $O(nm)$ sum, which sounds like it can get expensive, and there’s a product of $(n-1)$ probabilities, which sounds like it can get awfully numerically unstable. I’m surprised the authors have been able to train models at all. Are there any tricks involved? The appendix is incomplete. In fact, part of the proposed contribution of the paper is to provide a combination method for diffusion methods. This is never quite explained properly; readers are directed to appendix D which is empty. In addition, although the propositions and theorems are analogs of past work, it was quite surprising to find theorems with no proofs in a paper. Even if the proof is almost identical to prior work, reproducing it with proper credit seems like the least thing to do; in this case there are nuances with the GFlowNet framework that are left unexplained. The authors already highlight this limitation, but this seems like something quite fundamental that is for some reason not reported: **How expensive is it to train the classifier**? If it’s just as expensive as training a new generative model, then there’s little incentive to use the proposed method. I’d like not to have to take the authors’ word for it and instead see empirical evidence, training curves, wall time, and so on. Another missing result is some validation that the learned distributions are the expected ones (e.g. a plot showing that the JS divergence goes to 0 with more training/capacity) at least on a toy setting like the grid environment. I really appreciate what the paper is trying to accomplish, and the empirical results seem very nice (although lacking some crucial results). I’m not sure though that the proposed method is correct, and with no proofs or extra details of why this should work I’m really inclined to reject this paper. Happy to engage in conversation with the authors of course.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The authors have addressed some of the limitations of their work.

Response to rebuttal

Thank you for your response and apologies for the delayed response! > However, despite the known connection, classifier guidance for GFlowNets has not been proposed in prior work as far as we know. I agree, however I believe it would useful to highlight this connection in the paper. > Figures R.1, R.2, and R.3 show the cross-entropy loss of the classifier Thanks for these additional experiments. As you pointed out later in your rebuttal - there doesn't seem to be much improvement after a few hundred steps which is a bit surprising. Additionally, this makes it hard to (empirically) understand the effect of the classifier performance since we there isn't much range captured by the experiment. > Efficiency of training the classifiers Thanks for these details and the clarification. The training cost of the classifier is similar to training the GFlowNet from scratch. This is something which should be clarified in the paper, along with the computational challenges of using it with larger and more sophisticated diffusion models. I appreciate the authors response which answered most of my questions. I still believe the weaknesses remain, but the paper is strong enough for acceptance.

Response

Thank you for the answers! I have no further questions and maintain my positive assessment of the paper. > We understand the first part of the question in the following way: “The proposed method can easily turn a collection of GFlowNets for different objectives into a GFlowNet for any convex combination of the objectives”. Please let us know if our understanding is correct. Correct, my mistake.