A Generative Model of Symmetry Transformations

Summary

The paper aims to model the data distribution along each group action orbit. The proposed two-stage method first uses a self-supervised loss to learn an invariant function that maps each sample to its prototype and then uses a normalizing flow to learn the distribution along each orbit (i.e. conditional distribution on each prototype). To acquire the full data distribution, one trains a generative model on the prototypes and composes it with the conditional model. The method shows merits in terms of modeling the prototype-dependent distributions of symmetry transformations in the experiments on small-scale datasets.

Strengths

* The paper presents a novel framework for learning the distribution of symmetry transformations. Using a flow generative model allows for much more expressivity, compared to the parametrized Gaussian or uniform distributions considered in [1]. * The proposed method is easy to understand. The design choices are well-motivated and clearly described. * Experiments show interpretable results supporting the claims that * The proposed training objective can lead to approximately invariant prototypes. * The orbital distributions may vary for different prototypes. * The proposed method can increase the data efficiency for datasets with certain degrees of symmetry transformations.

Weaknesses

* My main concern is the practical applications of the method. Currently, the experiments are done on small image datasets, e.g. dSprites and MNIST. Can the authors identify some more complicated tasks where modeling the symmetry transformations could be beneficial? * Also, the dimensionality of symmetry transformations is generally much smaller than the dimensionality of the data manifold. For example, an image dataset may have a 28*28 pixel space and a lot of possible variations in there, while the symmetry of planar affine transformations only accounts for 6 dimensions of variations. Generative modeling is difficult because of the high dimensionality. I'm unsure if it's worth the effort to use a generative model to learn the low-dimensional distribution on a group orbit. * (This may be just my personal preference) I find the rotated and colored characters throughout the text a bit distracting. The normal texts have already made things pretty clear. Those characters may be great for intuition but are less accurate and formal. * Currently, the parameterizations for different symmetry transformations seem ad-hoc. E.g. affine transformations are represented in the affine matrix. However, symmetry groups have different structures (which can, for example, be reflected by the structure constants of the Lie algebra) and require different ways of parametrizing distributions. The authors should address this aspect and possibly discuss some related works, e.g. [2].

Questions

* Regarding the experiment setting, currently the MNIST dataset is manually transformed. I'd expect smaller variations in the original dataset. In that case, would the proposed method result in less performance increase? ## References [1] Benton, Gregory, et al. "Learning invariances in neural networks from training data." Advances in neural information processing systems 33 (2020): 17605-17616. [2] Falorsi, Luca, et al. "Reparameterizing distributions on lie groups." The 22nd International Conference on Artificial Intelligence and Statistics. PMLR, 2019.

Rating

6

Confidence

5

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors have discussed the limitations.

Summary

This paper proposes a Symmetry-aware Generative Model (SGM), aiming to learn (approximate) symmetry presented in a data. The model achieves this by mapping each sample onto a prototype—a unique representative on the group orbit—and learning the conditional distribution over its group orbit through maximum likelihood estimation.

Strengths

The paper is easy to follow

Weaknesses

1. Initially, I thought the paper aimed to address data-efficient learning of distributions with unknown approximate group symmetry. However, the scope of the paper is quite limited. Essentially, it only aims to learn the conditional distribution of symmetry transformations over the group orbits. In other words, the paper's primary focus is on augmenting training samples over their group orbits according to this supposedly accurate conditional distribution. 2. To demonstrate that the model is functioning as intended, the paper should provide results showing whether this conditional distribution is learned correctly. Unfortunately, this is not shown in any of the examples. The "realistic-looking" generated samples depicted in the figures are merely group-transformed versions of the given samples, which is why they appear realistic. 3. One of the main components of the paper is projecting each sample onto a unique representative of its group orbit, achieved through a so-called transformation inference function \( f_w: X \to H \), where \( H \) is the (potentially large) group. The paper trains this function \( f_w \) in a self-supervised manner through equation (6) to produce unique prototypes. However, this approach is completely unnecessary, as any equivariant function \( f_w: X \to H \) should already accomplish this. 4. The authors claim that one advantage of their approach is handling data sets with unknown symmetries. However, the paper only deals with (2D) rotation and scaling, as well as color transformations, which limits its generalizability. 5. Additionally, the images used in the paper, such as MNIST, are always compactly supported and vanish at the boundary. This makes data augmentation using rotation and scaling feasible. It is unclear how the proposed method would fare when dealing with realistic images that exhibit boundary effects after symmetry transformations. 6. The combination of the proposed model (which supposedly "learns" how to augment data) with a VAE is also unconvincing. While it might be better than directly applying data augmentation (as small images can become even smaller), without demonstrating that the conditional distribution is learned accurately, this combined model could very well learn an incorrectly symmetrized distribution.

Questions

Please see the above section

Rating

4

Confidence

5

Soundness

3

Presentation

3

Contribution

2

Limitations

Yes

Summary

The paper proposes a generative model that disentangles the latent space into a group-invariant part -- the latent for the prototype -- and another part which represent a group element that can be applied to the prototype to reconstruct the input. A key novelty is to simultaneously learn to predict a distribution for each input over the group elements based on data. This input-dependent distribution, in principle, can be then used to generate new data that better aligns with the true group distribution.

Strengths

The overall architecture seems interesting and sound and the idea of learning probability distribution over the group elements is especially interesting. The results show that the distributions are dependent on the input and are meaningful for the datasets used in the experiments.

Weaknesses

Some parts of the paper are confusing to me. What is the architecture for the part that predicts the distribution over the group elements? I saw one or two mentions of normalizing flows, but that is not enough to understand the details. The figure is quite unclear about it. Shouldn't the input image be also an input to the network that predicts $p_\psi(\eta|x)$? Also, the paper does not seem to have any information on the loss function used to train the network that predicts $p_\psi(\eta|x)$. A lot more clarity is needed for understanding these important details. I feel that the exact new contributions of the paper are a little unclear given many previous works doing similar things. Many of them are also mentioned by the authors in the appendix, but the contrasts with this paper are not clear. My understanding is that it is a generative model and can find the right distribution over the group elements directly from data, but there are earlier works also looking into these aspects. Some discussion here would be useful. Xu et al., Group Equivariant Subsampling Romero and Lohit, Learning Equivariances and Partial Equivariances from Data Shu et al., ,Deforming autoencoders: Unsupervised disentangling of shape and appearance Chau et al., Disentangling images with Lie group transformations and sparse coding Even earlier work by Grenander, Mumford and others on Pattern Theory discusses generative models and probability distributions over groups, but these are not based on neural networks. I think there should be at least one dataset which is a little more challenging like an image recognition dataset of more natural images. This can also help in understanding the limitations of the method. Another experiment showing the usefulness of the generative model being able to generate samples that respect the data distribution is also going to be useful.

Questions

Please address the weaknesses I have listed above.

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

2

Limitations

I don't think the paper mentions the limitations explicitly, which the authors should try to.

Summary

The paper proposes a generative model of symmetry transformations. The work leverages recent parameterizations based on group theory to define a generative model, in which relaxed symmetry becomes a latent variable over which inference can be performed.

Strengths

The paper is very well written and proposes an elegant method that leverages group theoretical framework to mathematically define approximate equivariances, combined with a probabilistic approach to guide discovery of symmetry transformations and learn a prototype. Lastly, the reviewer notes that the paper is very well-written and has beautiful illustrations which guide the intuitive understanding of the generative model and the concept around learning a prototype. Practicality It seems that the proposed SSL objective has computational benefits (easy to scale) as well as benefits in performance (ELBO objective only worked for rotations). Is this true, or am I reading this too positively? What would be potential disadvantage of choosing such loss? Experiments The paper provides very thorough experimental validation on a wide range of datasets. The appendices provide extensive details on the experimental set-ups making them highly reproducible.

Weaknesses

Intuition behind the objective. Experimentally the paper demonstrates that the proposed SSL objective is very effective. Apart from the computational benefits of such approach, it also seems to improve overall performance (optimizing ELBO only worked for rotations). To me it is not entirely clear why this would be the case, and it would be interesting to provide some more explanations on this - if known, of course. The first appendix was very helpful in providing context in relation to directly optimizing the ELBO. Types of transformations. The paper does not provide a lot of discussion on how the density of transformations is parameterized. In case of rotations, how is the normalizing flow constrained to remain smooth in the Lie algebra? Is the approach mostly targeted to simple (e.g. affine) groups?

Questions

1. The connection in App. A. offers a nice connection to the objectives used in some of the prior work. It references earlier approaches by the authors [Authors, 20XX] in which the ELBO is directly optimized and hypothesizes that ‘ averaging of many latent codes makes it difficult to learn an invariant representation a without throwing away all the information in x’ . Could the authors elaborate a bit more on this hypothesis / is this backed by any experiment? 2. Number of samples. The x-mse in number of samples is optimal for 5 samples. Don’t we expect this table to be monotonically decreasing in number of samples? 3. Overfitting on p(\eta | x). For very flexible distributions of transformations, isn’t there a risk of overfitting on the parameterization? Please correct me if I have missed something in the method which counteracts this.

Rating

8

Confidence

4

Soundness

3

Presentation

4

Contribution

4

Limitations

The paper offers a strong contribution proposing a probabilistic generative model that describes that as coming from transformed latent prototypes.

Brief discussion of additional related work for reviewer TzHS

*Group Equivariant Subsampling*. This paper introduces group-equivariant sub/upsampling layers. These layers can be used to construct group-equivariant CNNs with pooling-like layers. Our paper does not develop any novel architectures for group-equivariance, though our inference network could be made equivariant, as discussed in “Invariance of fω and the prototypes”. Thus, this line of work is related but orthogonal to ours. *Learning Equivariances and Partial Equivariances from Data*. Like ours, this work focuses on learning partial symmetries from data. However, this work is more closely related to that of Nalisnick and Smyth, [2018], van der Wilk et al. [2018], Benton et al. [2020], Schwöbel et al. [2021], van der Ouderaa and van der Wilk [2022], Rommel et al. [2022], Immer et al. [2022, 2023], Miao et al. [2023], and Mlodozeniec et al. [2023], in that it is concerned with the supervised learning regime. *Deforming autoencoders: Unsupervised disentangling of shape and appearance*. Like ours, this paper focuses on unsupervised learning of prototypical objects (called ‘templates’ in their work) and transformations to convert such prototypes into observations. However, our work is more general, being applicable to a more general class of transformations. Conversely, theirs includes many interesting inductive biases specific to shape transformations. Their method is not probabilistic, meaning that they are unable to sample novel observations. *Disentangling images with Lie group transformations and sparse coding*. Like ours, this paper learns to disentangle images into prototypes and their transformations. However, theirs differs from ours in two key ways. Firstly, they focus only on transformations corresponding to Lie groups whereas ours is more general. Secondly, they learn prototypes that are themselves composed of sparse combinations of learned dictionary elements. We see this work as complementary to ours, as the core idea of combining sparse coding with learned transformations could also be applied to our method.

Thank you for the response

The responses are indeed helpful in understanding some parts of the paper that I didn't before. I understand that the architecture details and other training details are in the appendix. Yes, it would be helpful if some of the important details are moved to the main paper. I understand the connection to existing work better. I still think having one dataset with more challenging images is important. At the very least, it shows what the limitations of the method are. Even a dataset like CIFAR-100 with 32 x 32 color images or CUB-200-2011 or PatchCamelyon would be interesting to show some experimental results on. A separate paragraph on limitations would be good to have rather than a footnote. For example, the authors say that running their method on more challenging datasets is computationally difficult. I don't understand if this is because the method is inherently more complex because of self-supervised learning objectives etc. or some computational constraints the authors may face. This is not too important, but this is one reference for Pattern Theory: Pattern Theory, the Stochastic Analysis of Real World Signals by Mumford and Desolneux. But the authors should not feel compelled to include this is in the paper if they don't believe it to be very related. Overall, I think the reviewers have addressed my concerns to some extent. I think it is important to have some experimental validation of more challenging datasets. Based on the above, I will raise my score to a weak accept.

End of discussion period summary

Thank you to all of the reviewers for engaging with our rebuttal and the discussion period. We are very happy that two of the reviewers (TzHS and d33o) have raised their scores, that the other two reviewers (rf86 and GGon) maintain their recommendations to accept the paper, and that *the scores now lean towards an accept on average*. We'd like to highlight a few of the comments that came up during the discussion period: * "I regard this a **technically strong paper, with novel ideas and a good execution**, and therefore keep my recommendation for acceptance with score rating 8." – Reviewer rf86 * "... **my main concern about demonstrating the correct conditional distribution learning [...] has been addressed**, I will substantially raise the score..." – Reviewer d33o * "**Most of my concerns have been addressed**... Overall, I maintain my opinion that this is a **well-written paper with clear motivation and reasonable experiment results**." – Reviewer GGon We believe that these comments highlight that the core concerns of the reviewers have been addressed, and that the reviewers see our paper as publication worthy due to it's novelty, technical strength, and strong presentation. The main remaining request is to include experiments on another (more complicated) dataset. While we haven't been able to do so due to the tight timelines and compute limitations in this discussion period, we will aim to include experiments based on the suggestion of reviewer TzHS in the camera ready version.