Summary
The paper pertains to the topic of data-driven symmetry discovery. The authors propose a method allowing symmetry discovery beyond pre-defined Lie groups, by learning to transform datapoints, potentially in a non-affine manner, via a learned ODE (referred to as the *one-parameter group*, where the single parameter is the time variable in the ODE), the velocity field of which is typically parametrised by an MLP.
Crucially, to optimise this, the authors choose an objective - *validity score*, which is a predefined measure of the extent to which a transformed datapoint is symmetric to the input one (in their examples: for images, they use the cosine similarity between features extracted by a pretrained NN, while for PDEs, they measure the value – error – of the PDE for the transformed datapoint). Additional regularisers are used to ensure the diversity of the symmetries learned (orthogonality between different learned velocity fields) and smoothness (minimisation of an estimate of their local Lipschitz constants). Experimentally, the method is tested on image classification (CIFAR10) and PDE solving (KdV, KS, Burger’s equations) showing that known symmetries are retrieved along with additional approximate symmetries, while the learned symmetries are subsequently used for data augmentation showing competitive results to methods using pre-defined augmentations.
Strengths
**Significance** . The paper studies an important problem (*data-driven symmetry discovery*) in machine learning, but also physical sciences where symmetries are abundant but potentially unknown. Identifying unknown symmetries and incorporating them in downstream ML models (e.g. via augmentation) can improve generalisation, especially in low-data regimes, while additionally, it can potentially provide novel insights about the task at hand.
**Novelty/Generality**
- The presented methodology has the capacity to recover symmetries arising from *non-affine* data transformations. This is contrary to prior work, where mostly linear/affine transformations are dealt with.
- Additionally, this method does not require making assumptions about the structure of the target group. This is common in prior art, where typically a subgroup of a predefined group is learnt.
- The authors take advantage of well-established concepts that are underexplored by the ML community (e.g. modelling transformations via the one-parameter group) - this helps to broaden the available toolbox in the field of ML & symmetries/ equivariant ML.
**Execution/Implementation**
- Although the proposed method has multiple complicated components (NeuralODEs, difficult objective to optimise for), it is nonetheless well-executed yielding competitive results and recovering known symmetries in popular testbeds.
Weaknesses
**Applicability and scope**. Perhaps the biggest limitation of the proposed method is the *reliance on the validity score*. Although the authors claim to be able to learn symmetries by making as few assumptions as possible (see strengths), this seems to be contradicted by the need to manually design a validity score. Moreover, I have the impression that the validity score is not merely a hyperparameter, but it is decisive for the symmetries that will be learnt (basically it is the objective function of the optimisation problem).
- For example, in the case of images, the choice seems ad hoc (closeness in the representation space of a pre-trained encoder NN). What leads the authors to believe that the features of equivalent (symmetric) images extracted from the pre-trained NN should be close? Have the authors tried to verify this assumption? I think the empirical validation is insufficient here (section 5.1.), so I am not entirely convinced.
- In general, I do not see a generic way to define validity scores and perhaps the authors have slightly overclaimed in that respect. I would like to read the authors' viewpoints on that. For PDEs, the validity scores are indeed reasonable and generic, so perhaps, they would like to put more emphasis on this perspective.
Furthermore, the authors introduce the concept of learning symmetries via the one-parameter group, claiming that it is more general than prior parametrisations that can only learn linear/affine groups. However, it is unclear what the present parameterisation can express, e.g. does it allow learning any continuous group or implicit assumptions are made here as well?
- Additionally, could the authors discuss if it would be possible to learn finite groups with this method as well and if not, how could those be incorporated?
**Related Work/Comparisons**. The work of Forestano et al., MLST’2023 is quite relevant to the present manuscript, with the main difference being that in that work, the transformations are parameterised by an MLP instead of a NeuralODE (the oracle used in this work seems similar to the validity score used here). Since the two works have many similarities, I think that the authors should discuss in more detail their differences and the advantages of their work (e.g. as far as I understand the MLP cannot guarantee that the transformations form a group). Note that modelling transformation via an MLP (or any NN in general) instead of a NeuralODE seems more straightforward and probably easier to train and more computationally friendly.
**Experiments**. I believe some additional empirical evidence would strengthen the authors' claims.
- Most importantly, an experimental comparison against the type of parameterisation used in Forestano et al. (MLP) should be provided, to verify if NeuralODEs are indeed a more appropriate parameterisation.
- Moreover, baselines are mostly missing, e.g. comparing against other methods for data-driven symmetry discovery (I am not super familiar with these works, but if I am not mistaken LieGAN by Yang et al., ICML’23 is a recent example).
- The reported results after augmenting with the learned symmetries do not seem to improve significantly compared to known/default augmentations. Can the authors discuss why this might be the case? This is important since it might undermine the necessity of the proposed approach. To be more convincing, perhaps the authors should perform experiments on problems where the symmetries are not known a priori.
- Additionally, ablation studies seem to provide important insights but are only discussed in the appendix. I would recommend being more upfront in the main paper and discussing in the rebuttal the following: sensitivity to hyperparameters (multiple are needed: loss coefficients, $\sigma$ and $\tau$), the method for choosing them, the difficulty of optimisation (3 terms are used in the objective) and if all losses are optimised in a balanced manner. Similarly, for the parameter $N_sym$, which is now chosen based on prior knowledge of the number of existing symmetries.
**Presentation/Exposition**. (disclaimer - this is a minor weakness) Given that the notions discussed here are probably not widely known across the ML community, I believe that the authors should aim to provide more in-depth explanations to make their work more accessible. For example,
- Multiple group theory/symmetry concepts are discussed without definitions (group generators, Lie group, Lie algebra, Lie bracket etc.). Additional examples that are not well-contextualised include in section 2 the one-parameter group discussion, the Lie algebra of the affine group and the discussion on the PDE symmetries (Lie point symmetries etc.). Adding some references here and providing some examples for the mentioned PDE symmetries would help.
- In section 5.2., some concepts regarding the experimental details are mentioned without appropriate explanations, while others are only mentioned in the appendix, although it appears that they are crucial for the method. Perhaps the authors should be more upfront and explanatory regarding the aforementioned.
Questions
- How is the weight function in L234 defined, and how important is this for optimisation? Are different weight functions ablated?
- It’s unclear why the stop-gradient is needed in L237. I did not find the justification fully convincing. Could the authors elaborate? What happens experimentally if one does not use stop-gradient?
- Although intuitive, it’s unclear why the inline Eq. in L212 holds for negative $\alpha$.
**Suggestion**.
In case the authors do want to present their method as generic, I think a deeper experimental evaluation in data beyond PDEs would help a lot (e.g. testing on other image datasets, ablating different validity scores etc).
**Minor**:
- What if the chosen validity score is not differentiable?
- *Notation*.
- The notation $\theta^V_s(x)$ is a bit dense (why use V as a superscript?). Eq (1) is a bit confusing. The inline Eq in line 83 sees clearer to me.
- Notation in sec. 4 could be also simplified a bit or made more accessible with examples (e.g. 4.1: give an example for $\mathcal{A}$, i.e. the set of all natural images).
Limitations
Some of the limitations are adequately discussed. An important missing point is, in my opinion, the need to manually design the validity score in domains beyond PDEs.
No foreseeable negative societal impact.