Learning from Pattern Completion: Self-supervised Controllable Generation

Summary

Conditional generation in the era of diffusion models has been positively impacted by ControlNets, which allows for the fine-tuning of diffusion models using additional image input allowing for fine-grain conditioning. Popular ControlNets rely on pose, edge, or segmentation map conditioning which requires additional, usually supervised, networks to derive the condition of interest. In this paper, the authors propose a fully unsupervised alternative to ControlNets and position their contribution as being proficient by mimicking the brain's modularity. They propose the training of an equivariant auto-encoder (SCG) model which allows for the disentanglement ("modularisation") of the input image information thereby allowing one to naturally access an altered version of an image that can serve as a condition for the generation in a ControlNet framework. Authors experimentally show the performance of the approach by 1) showing images conditioned on various representation modules, 2) comparing it against a Cany edge ControlNet, and highlighting how the approach is more robust to noise and produces more realistic images that resemble the input condition more closely.

Strengths

- SCG is a *fully unsupervised* method, which in contrast to most ControlNets, does not require relying on supervised tools - SCG allows for the generation of *various conditioning inputs* by allowing one to select one out of a set of representation modules. This is fundamentally appealing as ideally, it allows the user to select what part of the input information to condition the generation on instead of being tied to a set of (supervised) tools for information extraction in a classic ControlNet setup.

Weaknesses

I believe this paper is not in a publishable state for the following reasons: - Presentation/Clarity/Rigor: the paper should be polished as it contains a lot of typos, confusing sentences/sections (see below), and missing information (see questions). In general, sections 1 and 3 are hard to follow and cause some confusion, and figures - including captions - should be refined. - Impact: While the general goal of this paper (unsupervised + modular image conditioning) is appealing, in practice, SCG becomes a trade-off between supervision and control over the information that will condition the image generation. Standard ControlNets require supervised tools but allow for explicit control over the information that is embedded into the conditioning (e.g., body pose, and structural information through Canny Edges). SCG on the other hand, does not require supervision but results in a set of image conditions that mostly disentangles the input image according to spatial frequencies as pointed out by the authors. As a consequence, most images capture similar semantic information and generated images are very similar to one another. To summarize, I believe the current use cases of SCG are cases where one wants to generate novel images that slightly differ from the input image condition, similar to using a ControlNet on Canny Edges (which is indeed the only ControlNet that authors compare against). I think this should be made more clear in the manuscript from the get-go (only discussed in the limitations) and ideally the paper should address this concern to increase the impact of SCG. - Experimental validation: I believe the experimental validation does not allow the reader to have an in-depth understanding of the performance of SCG. Mainly the following questions are unanswered: how much variability (in quantitative terms) can one expect from conditioning on various modules? how do images generated from SCG compare with images generated by conditioning on the lower principal components (see [1] for a more detailed example)? [1] Balestriero, Randall, and Yann LeCun. "Learning by reconstruction produces uninformative features for perception." arXiv preprint arXiv:2402.11337 (2024). Minor: - section 1: overall confusing, I believe the important message here is "brain performs functional modularity, we aim at mimicking this for associative generation purposes; As a starting point, we believe equivariance helps in achieving functional modularity in neural networks". The current state of the introduction is convoluted and it takes some thought to actually extract the general message. - line 33-36: should be rephrased in my opinion, the sentence is confusing/hard to follow. - line 43: are we talking about the brain or neural networks here? this sentence is confusing. - line 45: problem in the sentence formulation - line 53: I believe "devide" should be changed to "divide" throughout the paper - line 109: loose terms $\rightarrow$ "is a kind of change" - line 120: confusing formulation "train the autoencoder in homologous images" - section 3: what is the fundamental difference between z, the latent representation and, f, the feature maps ? We seems to be switching back and forth between both notations. - section 3: why does the latent representation $z$ become a function (i.e., $z(I)$)? - Eq 6: sum over $\delta$, which set of values does $\delta$ belong to? - section 3: symmetry loss, the intuition behind this additional loss term is unclear to me. - line 138: typo - line 146: confusing, either ControlNets should be described in more detail in section 2 or the phrasing "based on ControlNets" should be clarified in section 3 to make it evident that the "pattern completion" is entirely mediated by the existing ControlNets framework and is not a contribution. - Eq 9: the expectation symbol should be replaced by \mathbb{E} - Figure 1: This figure is supposed to reflect the approach proposed by the authors, however, it is not straightforward which elements in this figure correspond to the SCG building blocks detailed in section 3. Instead of clarifying SCG, figure 1 brought some confusion in my case. - Figure 3: formulation problem in the caption - Figure 4: typo in the caption and the x-axis label

Questions

- can authors confirm that the conditioning used is $f$, the latent representation? - can you clarify what the symmetry loss terms aim to do? - can you explain what translation and rotation augmentations were used for the training of the autoencoder? what is the sensitivity of the representations to these parameters? - can you explain the training procedure of the autoencoder? number of epochs, optimizer, ... - can you confirm that the encoder and decoder "architectures" are single-layer convolutional filters? - how is the subject evaluation performed? how many annotators? through which annotation platform? on how many images? which questions are asked of annotators?

Rating

5

Confidence

4

Soundness

2

Presentation

1

Contribution

2

Limitations

The authors discuss the fact that the current equivariant autoencoder disentangles mostly based on spatial frequency rather than following a human-interpretable partitioning of the information (high-level/semantic vs. low-level information). Beyond that point which limits the impact of this work, I believe additional limitations such as the lack of control over the redundancy between the original image and the image condition is a limitations that is tied to the use of fully unsupervised despite clear advantages to choosing fully unsupervised methods.

Summary

The authors train a modular auto encoder with an auxiliary custom equivariance objective, which enables them to get independent sets of representations of an image. The authors then use some of these representations to condition a ControlNet. The authors find that the auto encoder’s submodules learn to encode functional specializations such as edge detection and other low-level image features.

Strengths

- This paper contributes an exciting and challenging field: controllable image generation. The work presented in the paper is original. - The authors make their code available with the submission. This makes the paper easier to review, and will greatly improve the paper’s usefulness for other research groups working on controllable generation.

Weaknesses

- The paper is titled “Learning from Pattern Completion”. What are we learning from pattern completion? - Section 3 should be written more clearly. - While the paper promises controllable image generation, it appears that the proposed method is more of an image reconstruction method: the experiments from the paper evaluate the proposed method by giving it features generated using the proposed modular autoencoder as input. My sense is that if I wanted to draw “a monkey climbing on the moon”, then I could give ControlNet a hand-drawn sketch of this, while the proposed SCG method would not be able to generate an image based on my sketch. Is this correct?

Questions

- The authors chose to evaluate ControlNet with the Canny image detector. Would a different representation to condition on have worked better? It seems obvious that the Canny image detector is a lossy representation of the original image. - The authors could use a tilde (~) before \cite commands to put a small space before the square brackets, which would make the text more visually pleasing. For example~\cite{schmidhuber_1994}. - I believe that instead of “equivariant constraint”, the correct term is “equivariance constraint”. - It would be great if the README.md of the code could state what kind of GPU and how much RAM is required to run their code. - It would be great if the authors could include more detail in the caption of Figure 2, especially for part C of the figure. - On line 56, it appears that a word is missing between “sparsely” and “in” - On line 66, the authors use the term “closed and complete independent manifold”. The terms “manifold”, “closed” and “complete” have specific meanings in the mathematical area of topology, and if I understand correctly the authors are not using the words in the sense of the definitions from topology. It may be useful to use different terms here to avoid confusion. - On line 93, the authors mention “simple cells”. It would be helpful for some readers to change this to “simple cells in the visual cortex”. - On line 130 it says “where M^(I)(delta) is a learnable prediction matrix determined by delta”. I still do not understand what M^(I)(delta) is. Is it a neural network that takes delta as input? Or is it computed for every delta? More generally: what is the set or distribution of all deltas? Equation (6) involves a sum over deltas, and I don’t know what the set of all deltas (or its distribution) is. - I would be grateful if the authors could describe equation (8) more. Is C able to construct z(I) based on z^(i)(I) and i alone? - On line 172, “We” should be lower-cased.

Rating

5

Confidence

3

Soundness

3

Presentation

3

Contribution

2

Limitations

Yes

Summary

This paper proposes a self-supervised controllable generation (SCG) framework with two training stages. The first stage exploits equivalence invariance to learn a modular autoencoder, aims to extract different visual pattens from input images, each extracted (learned) visual pattens can be regarded as a kind of image condition. In the second stage, perform as the standard ControlNet training with these self-supervised extracted image conditions. After self-supervised training, SCG surprisingly shows some zero-shot capabilities, which can show excellent generalization capabilities for various tasks such as associative generation of paintings, sketches, and ancient graffiti. The authors compare SCG with ControlNet in terms of SSIM/PSNR/CLIP-score and achieved competitive results. However, the authors lack further exploration of the self-supervised trained ControlNet. Overall, I like the idea and method very much, but I still have some doubts about the validity of the experimental results and the fairness of the comparison.

Strengths

1. I like the idea that exploring self-supervised training on controllable generation models. Compared with existing methods, the self-supervised training framework proposed in this paper does show better generalization ability to a certain extent. 2. Using the equivariant constraint to learn a modular autoencoder to extract different visual patterns is a simple, elegant and effective approach. 3. Although self-supervised training has not seen conditions such as painting, sketches, and ancient graffiti, it still shows good zero-shot generation capabilities, which verifies the rationality of training modular autoencoder with equivariant constraint. 4. Compared to previous excellent works ControlNet, SCG has a higher or similar winning rate in fidelity and a significantly higher winning rate in aesthetics.

Weaknesses

1. The MS-COCO dataset is used for both the self-supervised training phase and the evaluation phase, which may lead to a serious unfair comparison because the original ControlNet is not trained on MS-COCO. 2. Although this paper explores the self-supervised training of ControlNet and its zero-shot generalization ability, no further experiments are conducted, such as whether using it as pre-trained weights for further fine-tuning on labeled tasks such as depth, segmentation, etc. will bring further improvements of controllability? The zero-shot capability alone cannot fully demonstrate the effectiveness and necessity of the proposed self-supervised training. 3. Based on the above weakness, it is necessary to conduct ablation experiments on the number of modules (from modular vae) and the types of Equivariant Constraints. 4. In addition to the painting, sketches, and ancient graffiti mentioned in the paper, does SCG also have certain zero-shot capabilities under some other conditions such as Edge/LineArt/Segmentation?

Questions

Please refer to the weaknesses.

Rating

7

Confidence

4

Soundness

4

Presentation

4

Contribution

4

Limitations

NA

Summary

Presents a self-supervised approach for learning multiple distinct representations from images through a loss leading to specialized modules, each learning different aspects of the data without manual design. Demonstrate that those modules may be useful for the conditional generation of images through a controllable diffusion model.

Strengths

* Originality: the main contribution of the work is in the novel loss function and the insight (perhaps inspired by neuroscience) that modular networks are favourable for learning in self-supervised settings. The idea that a module may develop functional specialization from imposing equivariance to some property of the stimuli is also a nice extension of previous supervised approaches. I think the other half of the work, where the representation of a single module is used for controllable generation, is a natural extension of previous work. Previous works on both manual designing specialized blocks and on Group equivariance are properly quoted. * Quality: the presented experimental results are fine (but as always with image generation, visual judgement is difficult), so the quantitative evaluation (Table 1) and subjective evaluation (Figure 4 and some panels in Figure 7) are appreciated. * Clarity: the submission is clearly written and well organized, using neuroscience inspiration to improve the intuition on how to make progress on self-supervised learning of rich representations. The “pattern completion” jargon, on the other hand, does not contribute to the reader’s understanding; it would have been better to adhere to “conditional, controllable generation,” which actually describes what was done. * Significance: the suggested equivariance and symmetry loss components are easily usable for developing a functional specialization in other projects and may impact seemingly unrelated problems.

Weaknesses

* Quality: the evaluation is limited only to ControlNet, which is probably a decent baseline, but there are other approaches for image generation that can be compared with (e.g., it’s fine to show a manual-designed solution outperforming the current self-supervised model or compare with other self-supervised approaches). * Significance: it would be great to see if the suggested loss components can be utilized in a setting other than controllable generation where their contribution can be better quantified, e.g., in lowering MSE in a reconstruction or prediction task.

Questions

1. What is the respective contribution of the symmetry loss vs the equivariance loss? What does performance look like with only one of them? What is the problem solved by introducing each of them? 2. What are the equivariance properties of HC1 and HC3 (i.e., their functional specialization) and make them useful for controllable generation?

Rating

7

Confidence

3

Soundness

3

Presentation

4

Contribution

3

Limitations

There is only a limited discussion of the approach's limitations and the limits of functional specialization (summarized as “only low-level feature specialization”); it would have been nice to see a discussion of what useful features are NOT learned by this approach and why.

Response to rebuttal

I thank the authors for their clarifications and am satisfied with the improvements. I have adjusted my score accordingly.

Answer to rebuttal

Thank you to the authors for their answers. My concerns regarding the fact that the method allows for modulating elements like the contrast and brightness of the images but doesn't offer much variability over the semantics ( images are nearly identical to the condition image) therefore limiting the appeal of the approach, remains. Also, the authors do not answer my question regarding the comparison with conditioning the generation on images resulting from Principal Component filtering. The additional results provided by the authors are informative and show that for some more unusual styles, like graffiti, the conditional offered by the modular autoencoder outperforms supervised methods while it is not the case for more usual styles, where the images conditioned with segmentation mask look as good but with more variability in the image instead of reconstructing the condition image. Given additional results and after reading the other reviews, I see better the novelty of the proposed method and how it might inspire future work. For this reason, I am increasing my score.