A General Theory of Correct, Incorrect, and Extrinsic Equivariance

Summary

The authors analyze how equivariant models behave under mismatches between the symmetries of the model and data distribution. They advance three notions, correct, extrinsic, and incorrect pointwise equivariance. They then propose bounds on the error for equivariant neural networks that are sensitive to the data distribution and give an example where having an equivariant model can hurt learning performance. (Edit, 8/13: in light of comments by the Authors, I have changed my score to a 5)

Strengths

Theoretically, the bounds present are more general than those proposed in Wang et al. 52. The examples clearly demonstrate the underlying mathematical principles involved. Moreover the error bounds presented, while not unsurprising to practitioners more versed in group theory, seem mathematically solid. On the whole, the task of attempting to precisely characterize how the data distribution interacts with symmetries imposed on the model is a worthy one.

Weaknesses

While the paper is solid, I think certain difficulties in presentation prevent it from fulfilling its potential. Most immediately, I think there are some issues in the presentation in the concepts of pointwise correct / incorrect / extrinsic equivariance. First, equivariance is a property of maps rather than spaces: saying a point has correct / incorrect / extrinsic equivariance, or even equivariance of any kind, isn’t precise. It would be better to say something the lines of “h has correct / incorrect / extrinsic equivariance with respect to f and p at x”, particularly since the notion of correctness here is dependent on the model: this has the advantage of being a more close parallel with Defs 3.1-3.3. Second (forgive me if I missed it), but at no point is the notion of pointwise equivariance being dependent on specific group elements used in the rest of the paper. It is not clear what is gained by this added generality. Continuing on to Section 5, it is not clear to me to what extent the notions advanced in section 4 are necessary for stating these bounds: if I understand correctly the bounds presented are for general probability distributions and are independent of the definitions proposed in 4.1-4.3. Section 6 seems a bit trivial for me: it seems obvious to me that if you can’t represent a decision boundary due to a constraint in the model you might end up having a bad day. Indeed, the specific example shown here seems to follow straightforwardly from https://doi.org/10.48550/arXiv.2110.07472. I think this section should be moved up and be a motivating example for the bounds in Section 5. I feel like the meat of the paper is the error bounds, more specifically Theorem 5.8. I’m not convinced the concepts in Definitions 4.1-4.3 contribute much towards the construction of this bound. I would recommend a rewrite of the paper that focuses more on the bounds, their implications and applications, and maybe expand further. Finally, (and this might just be me being a pedant), but I’m not sure the title really is reflective of the content of the paper. It is not clear to me what the general theory is that is being advanced. Typically a “general theory” is a precise characterization of a class of mathematical objects. If this is intended to refer to the decomposition into correct / incorrect / extrinsic equivariance, then this characterization seems to reduce to the fact that that equalities can be true, not true, or involve the empty set. I don’t think any insight is gained innately from this characterization, rather than this being a useful language to describe further results in the text. The error bounds, while good contributions, are specific to certain choices of loss function. I mention this not to diminish their contributions, merely to observe that they are not “general” in a mathematical sense. (edited for some spelling mistakes, although I'm sure more of them abound...)

Questions

I’m having a little trouble parsing the notion of Fundamental domain in Definition 5.1. What happens if the orbit of the passes over X multiple times. For instance, consider the real line, where the group is S_8 and the group action is the antisymmetric representation (reflect the line on odd permutations)?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

2 fair

Presentation

2 fair

Contribution

2 fair

Limitations

The limitations of the work have been appropriately addressed.

Summary

This paper provides a general theory of correct, incorrect, and extrinsic equivariance of functions, mainly extending the framework of Wang et al., 2023 to more general case of pointwise equivariance of functions defined for pairs of group element and input data. The theory mainly concerns deriving lower bound of errors for classification and (invariant and equivariant) regression tasks. The authors further identify cases where extrinsic equivariance can be harmful for performance opposed to the empirical observations of Wang et al., 2023. The authors provide a range of experiments mainly on empirically verifying the derived lower bound of errors, and also demonstrating the cases where certain extrinsic equivariances can be harmful. Wang et al., The surprising effectiveness of equivariant models in domains with latent symmetry (2023)

Strengths

S1. The theory proposed in the paper is indeed quite general as it considers pointwise equivariance, and covers a large range of partial symmetries. The fact that it addresses a wide range of tasks (classification, invariant regression, equivariant regression) also strengthens the generality of the paper. S2. The paper is overall well written, with intuitive illustrations on pointwise equivariance and experimental setups, as well as results. S3. The experimental results support the main claims of the paper on error bounds and harmful cases of extrinsic equivariances.

Weaknesses

W1. A weakness of this work is that, while certain cases are presented where extrinsic equivariances can be harmful, it offers little principled understanding or general theory of in which specific cases extrinsic equivariance is harmful or beneficial. W2. While the presented theory on pointwise equivariance covers a wide range of approximate or misspecified symmetries, I am not sure if it is immediately useful, since in many applications involving approximately or misspecified equivariant neural networks, we would not be able to exactly know the extent of correct, incorrect, and extrinsic pointwise equivariance of the model (unlike the synthetic experimental setups considered in the paper). This could be a weakness of the proposed theoretical framework.

Questions

I have no particular questions, but would like to hear the opinions of the authors on the aforementioned weaknesses.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The authors have identified the limitation of the work in Section 8.

Summary

An obvious limitation of equivariant networks is the assumption that the symmetry they hard-code matches the symmetry of the underlying ground truth function exactly. What happens when the symmetry is only partially present in the domain affecting a symmetry mismatch between the ground-truth function and the neural network? This paper presents some systematic analysis of this situation building on prior work by Wang et al. 2023. Wang et al. classified models as having "correct" equivariance (the underlying function provided by nature matches the symmetry of the equivariant model), "incorrect" equivariance (there is a disagreement between the underlying true function and that encoded by the equivariant network), and "extrinsic" equivariance (model symmetry transforms in-distribution data to out-of-distribution data). This paper defines these notions pointwise which allows one to generalize to a continuum of equivariance types when some tasks may have different proportions of these three situations (and does not exclusively conform to only one of them). Lower bounds are provided on the model error from incorrect model symmetry (while improving some earlier results) for classification. Morally similar lower bounds are provided for the regression case in terms of the variance of the function to be modelled over the orbit of the group under consideration. It is also shown that extrinsic equivariance can be harmful (which doesn't contradict but can still be contrasted with earlier experimental results in the literature). The theoretical results are provided in section 5 (after first introducing the notions of correct, incorrect and equivariance of Wang et al, and then stating their pointwise generalizations which open the path to interpolate between them -- a central aim of the paper). The lower bound for classification is intuitive and basically says that it is equal to the integral of the total dissent over the fundamental domain. The total dissent measures how many elements in the orbit of G have a different label than the majority label. Two examples are given for binary and multiclass cases to build intuition (also illustrated in figures 3 and 4). Next, the case of invariant regression is treated. The result is similar in spirit to the result from earlier -- the error is bounded by an integral over the fundamental domain of p(Gx) times the variance of the function on the orbit Gx (so instead of the dissent, we look at the variance). A slightly more careful consideration permits an easy generalization for equivariant regression. A simple theoretical argument is then provided to show that extrinsic can be harmful for generalization, also raising the question that it remains open to understanding its effect on generalization.

Strengths

- The paper considers an important problem that has only recently started getting treated in the literature. It is clear that equivariant networks make a strong assumption about the symmetry of the ground truth function. However, it is not clear how much a mismatch matters. This paper provides some approximation lower bounds due to the mismatch for some different settings. - Being able to interpolate between the three types can definitely help improve model generalization and permit more flexible models. - While many of the points raised in the paper seem somewhat obvious at times, and computing the lower bound might not be possible in practice, the presented work can still give some guidance on model selection.

Weaknesses

- Somewhat fortuitously, I was a reviewer for this paper for an earlier iteration. Whatever weaknesses that I had raised at the point seem to have been adequately addressed by the authors (in fact they were right then). I also notice that the other weaknesses bought up by other reviewers at the time (mostly in terms of experiments, and comparison to the work of Wang) have also mostly been incorporated. While I can nitpick, I have no hesitation in simply voting for acceptance.

Questions

No

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Yes

Summary

The paper analyzes error bounds for models constrained to satisfy symmetries that only partially agree with the ground truth functions. The main suggestion is to generalize previous definitions, to the point level, allowing to derive a lower bound on the model error with respect to the volume of the portion of the domain where symmetries are mismatched. The theoretical analysis is validated on some toy examples and some (relatively) small-scale datasets

Strengths

The proposed analysis including the definitions, propositions, and theorems seems to be simple. The paper is well-written and easy to follow. I appreciate the illustrations and toy examples provided. The experiments seem to support the theoretical analysis. Where mismatches occur, reasonable explanations are provided.

Weaknesses

Missing discussion I feel the text should elaborate more on the assumptions taken in the analysis of error bound. For example, it is assumed that the model assigns labels by taking a majority vote in an orbit. How reasonable is this assumption in relation to existing equivariant models? In addition, it would be interesting to incorporate a model that approximates the majority vote by sampling for one of the experiments. E.g., does it improve the digit classification model?

Questions

No further questions.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

Yes

Summary

The paper presents an extension of a lower bound on error in finite labeled classification introduced in "The Surprising Effectiveness of Equivariant Models in Domains with Latent Symmetry" and investigates lower bounds on error for G-equivariant and G-invariant regression, offering valuable insights into the performance of symmetry-preserving models in regression tasks.

Strengths

This extension holds significant practical importance, particularly when considering real-world natural symmetry groups such as $G = SO(3)$, which go beyond the scope of the original paper. Moreover, the authors have done an excellent job in presenting the material in an accessible manner, making the paper easy to read and well-structured.

Weaknesses

1. The paper's focus on pointwise definitions rather than emphasizing their implications may obscure the novelty of the results, potentially hindering a clear understanding of the significance of their contributions. 2. While the experiments conducted in the paper offer valuable insights, their omission of infinite groups may limit the demonstration of the full strength of the new lower bound. Additionally, the observations suggesting equivariant models' potential ineffectiveness in classical learning tasks raise valid questions about their relevance and applicability. 3. Although the paper draws upon certain notions from "The Surprising Effectiveness of Equivariant Models in Domains with Latent Symmetry," it does so only partially. This partial utilization of concepts, coupled with the use of different evaluation metrics such as "accuracy" in the original paper and "error" in this work, may create challenges when attempting to directly compare the results between the two studies.

Questions

1. Can you elaborate on the third experiment? Which state is flipped? (line 532, page 9) 2. Did you mean "orbits" instead of conjugates in the definition of the fundamental domain? (line 158, page 4)

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

2 fair

Limitations

The author did address the limitations of reflection symmetries on the expressivity of models in Chapter 6 (line 254, page 7), and the requirement of knowledge about the density function (line 386, page 9)

Summary

This work addresses the situation in which data and model equivariance does not match exactly. It extends previous work from Wang et al., by proposing a pointwise version of their definition of correct, incorrect and extrinsic equivariance. The usefulness of these new concepts are demonstrated in three new performance lower-bounds derived from classification and regression problems. A series of examples and experiments are presented to illustrate the new proposed notions, confirm the lower-bounds derived and show that they usually seem tight in practice.

Strengths

### Originality Although it builds heavily on previous work from Wang [52], the paper presents novel theoretical concepts and results, as well as new experiments supporting them. ### Clarity Overall, the paper is well written and structured. ### Quality The paper presents new theoretical results with their corresponding proofs, which look sound to be (disclaimer: I have some familiarity with group theory, but I am not an expert). It also presents interesting results in a large range of experiments. The latter are all rather small/toy, but are still quite convincing IMO.

Weaknesses

### Clarity Minor: - I think there is a mistake in the xlabel of fig 7b. Shouldn't it be "incorrect - correct"? Same for figure 7a: should be "correct - extrinsic" I guess, since when x=1 you have the highest INV model performance, which should correspond to c=1 and e=0 according to the text. - L.363: I think you wanted to reference fig.9, not 8. ### Clarity and Quality 1. I think the definition of $p$ could be clarified. It is introduced in line 92 as the “probability density function of the domain”, so, at first reading, I thought p(x) was the “true” underlying population distribution from which both training and test examples are samples. This seems to be confirmed by line 387 in the conclusion: “our theoretical lower bounds require domain knowledge like the density function over the domain”. But in section 6, it seems that you define extrinsic equivariance with respect to the actual examples in the training set, which is not the same thing: “F_E corresponds to an extrinsically equivariant class for **this data**” l.272 2. I also have other few questions regarding the example presented in section 6. First of all, does the data from figure 5 represent training or test data? Is the data $S$ the whole support of $p(x)$? What is the true labeling function $f$ in this example? These elements seem important to conclude. The reason I ask is that if $f$ is indeed the “exclusive or” on (x,y) coordinates (i.e. it is indeed C2-invariant) and if the examples from figure 5a are just the training data but the test data can go outside (e.g. their symmetric elements in fig 5b could be the test set), then despite the 0% error rate of the unconstrained linear model on the training data, it would learn an incorrect labeling function and its test performance would be 50%, while the invariant model would still have a 25% error rate, which would be better. 3. If $p$ really denotes the population distribution, I wonder whether we can talk about $p$ independently of $f$ and vice-versa. From your figures 1 and 2, it seems that $f$ can be defined outside of the support of $p(x)$, and I wonder whether this makes sense. For example, if $f$ is the labeling function in a digit classification problem, what should be its output for an image of the digit “9” rotated by 90 degrees? Should it be 9 or 6? ### Originality: Minor: 4. In the related work, some references to class-specific and instance-specific automatic data augmentation works are missing ([1, 2, 3] for example), while they are strongly related to the idea of pointwise invariance proposed in the paper. [1] https://arxiv.org/abs/2106.13695 [2] https://arxiv.org/abs/1510.02795 [3] https://arxiv.org/abs/2206.00051

Questions

See 3 questions in the weaknesses section above.

Rating

8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Yes, the authors mention one limitation of their work, which is the need to know $p$, which is never the case in practice.

Answer to authors

Dear authors, Thank you very much for your thoughtful rebuttal. I can confidently say you have addressed all my (few) concerns. The only points I would like to slightly insist on after reading your clarifications would be: 1. I think indeed that it would be nice to make clear that $S$ is the whole support from which any data can be sampled in section 6 (even though the example is very simple) ; 2. I also think you should say explicitly what is $p$ and that you assume there is never a distribution shift (so $p$ is always the underlying distribution when training **and** testing). I understand now why you have drawn $f$ outside of its domain, as your figure would indeed be less clear if you had restricted it to $p(x>0)$.

Response to the Response

Thank you to the authors for their detailed response to my comment. I still have reservations, but in light of proposed changes and the feedback of other reviewers, I am happy to raise my score to a 5. A lot of the nature of the feedback is related to the question of, "how straightforward or how non-obvious are certain elements." I'll discuss this more in my response the other reviewers above. There are two technical points I wanted to follow up on. > The pointwise equivariance is actually used in Proposition 5.4 as a way of calculating the error bound for the classification task. Later on, we also used it in Example 5.5 and the experiment in Section 7.1. The pointwise definitions are necessary for understanding our lower bounds because, intuitively, the bounds calculate the error of the model where pointwise incorrect equivariance exists. There are two issues at play here. The first is, "is p-wise equivariance necessary *in section 5*" It definitely is not in proposition 5.4 (which holds far an arbitrary density), and I maintain that ex. 5.5 could easily be rewritten to not use these definitions. This is not a criticism of the contribution of the bounds though: rather, it is me saying that the bounds in Section 5 are actually general for *all* probability densities, not just for incorrect equivariance as stated in the paper. My proposed (but only suggested) change is that the authors be bolder at selling the bounds, as they are a separate co-equal contribution to the theoretical concepts in section 3: currently, I think and importance of the bounds is undersold. The second is, "at any point in the paper, is the fact that of p-wise equivariance depends *on a specific group element $g$* used? As far I can tell it is not: only the dependence on $x$ is used, even in the examples mentioned by the authors. Consequently, it should be possible to make more specific definitions that only have dependence on $x$. However, I recognize this might be more of a style thing, and changing the definition at this point may require rewrites). > Generally, if the orbit passes.... Ok, this is how I thought the fundamental domain worked. But in this case, isn't the assumption that intersection of any two conjugates have measure 0 violated? As far as I can tell, the authors use the word "conjugate" to mean a set as $g F$ or some $g$ in the group. (I might be wrong about this: the use of conjugate in this paper is not the typical one in group theory, and I don't see a definition in the paper). In the example we are discussing, the conjugate of the group elements $e$ and of any even permutation have as their conjugate {$x \geq 0 | x \in R$}. So, by {assumption $x \geq 0 | x \in R$} has measure 0. By similar arguments, the same is true for {$x < 0 | x \in R$}. But this suggests that, by countable additivity, the entire real line has zero measure, which seems like a problem to me. Did I misunderstand something? If not, their use of fundamental domain could noticeably restrict the applicability of their results, but I think this could be fixed without too much effort by modifying the non-intersection assumption to only hold for coset representatives for cosets of the stabilizer subgroup. *edit: I just saw the Author's response above giving their definition of conjugate, which makes me more confident in the analysis above.

Response to rebuttal

Thank you for the response. I recommend the authors to supplement the discussions on limitations in Section 8 with the provided response. I have no further questions for now.