Summary
This paper provides theoretical investigation on the intuition that, machine learning models (e.g. neural networks) whose symmetry aligns with the data symmetry achieve best performance and generalization. The paper formulate the relationship between model equivariance error and data equivariance error by providing quantitative bounds, which are applicable in the presence of approximate or partial data/model equivariance.
Strengths
1. To my best knowledge, the paper provides a systematic and comprehensive review of the historical development of neural networks with group symmetries (**since this is not exactly my area of research, please correct me if I'm wrong**).
2. The paper provides detailed and systematic analyses of the model generalization bounds as well as approximation bounds under the more general case of partial or approximate equivariance. According to the authors, as well as to my best knowledge, this is the first theoretical work that addresses this problem. Therefore its contribution and novelty is significant.
3. Although not being able to check every technical details and derivations, I find the paper well-structured and easy to follow.
Weaknesses
**Major issues:**
1. Derivation of the approximation error bounds:
- First of all, in line 291, I don't think it's legal to write $\min A - \min B$ as $\min(A-B)$, since the second term has a minus sign. So I think the correct way to write is that
$\text{AppGap}(\mathcal{F}, \mathcal{D}) := \underset{f\in\mathcal{F}}{\min}\mathbb{E}[\mathbb{E}_g[f(g\cdot Z)]] - \underset{F\in\mathcal{M'}}{\min}\mathbb{E}[F(Z)] \ge \underset{F\in\mathcal{M'}}{\min}\mathbb{E}[\mathbb{E}_g[F(g\cdot Z)]] - \underset{F\in\mathcal{M'}}{\min}\mathbb{E}[F(Z)] $
- The expression above provides a **lower bound** to the approximation gap, so how does the inequality become equality in Line 304? If it is derived by assuming $\mathcal{F} = (\mathcal{M}^{(G)})'$ (assumption 3 in line 301=303), then line 304 should be written as
$\text{AppGap}((\mathcal{M}^{(G)})',\mathcal{D}) = \underset{F\in(\mathcal{M}^{(G)})'}{\min}\mathbb{E}_Z[\mathbb{E}_g[F(g\cdot Z)]]$
- Moreover, if the upperbound in Prop 5.1. is derived based on the equation above, I think it's necessary to prove that $(\mathcal{M}^{(G)})'\subseteq \mathcal{C}_{\epsilon,\lambda}$ , for any $\epsilon\ge 0, \lambda\in(0, 1]$, which I don't see in the proof.
2. Interpretation of the "generalization-approximation trade-off"/"optimal equivariance" is not clear:
In line 360-362, it is claimed that the optimal $\lambda^*$ "validates the intuition stated at the onset that for the best model its symmetries must align with that of the data". However, based on my understanding, the symmetries of the model **only fully align with those of the data** when $\lambda^*=1$, given $\epsilon=0$. When $0<\lambda^*<1$, I think it means the model only **partially preserves the symmetries of the data**. Do you agree? If this is the case, how do you justify the intuition that "the integration of relevant symmetry results in enhanced generalization."?
3. Similar to reference 9 in the paper, I think one key point I would expect to see in this paper is proving that, models with aligned group symmetries of data exhibit higher sample efficiency/complexity, which can generalize to scenarios of partial/approximate symmetry. I see no discussion or proofs regarding this intuition in the paper.
**I will be happy to raise my score given satisfactory explanation or revision.**
**Minor Issues:**
1. This paper targets specifically on the theory of machine learning models with partial/approximate group symmetries, which might be of interests only to a very specific subgroup of the whole machine learning community.
2. There are some presentation issues need to be improved:
- All hyperlinks do not work when clicked. Some numberings are also confusing, e.g. what is Thm A.4 in line 242, 246, 268?
- Some of the statements would benefit from more citations/examples, e.g.
"It has often been noted that this may impose an overly stringent constraint as real-world data seldom exhibits perfect symmetry” (line 36 - 37)
3. Typos
- may have not -> may not have (line 312)
- $\text{Stab}{\epsilon}(\mathcal{F'}) \Longrightarrow \text{Stab}{\epsilon}(\mathcal{F})$ (line 309)
Questions
1. More regarding the interpretation of the "generalization-approximation trade-off"/"optimal equivariance":
If we examine the contribution of $\lambda$ to the generalization and approximation bounds given by Theorem 4.4 and 5.2 respectively, assuming perfect symmetry of data ($\epsilon=0$), we see that the former decreases when $\lambda$ increases ($\frac{1}{\lambda^{1/2}}$ or $\frac{1}{\lambda^{1/d_0}}$ scaling), while the latter increases when when $\lambda$ increases ($\lambda^{1/d_G}$ scaling), which gives what the authors called "generalization-approximation trade-off". To me, this seems to show that minimizing the generalization error effectively aligns the model and data symmetries, whereas minimizing the approximation error functions in the opposite direction, resulting in a partially aligned model symmetry.
Do you agree with this analysis? Any further explanation/intuition?
2. Can you explain what you mean by "oscillations", which is mentioned repeatedly? Example contexts like:
- "since $\text{Stab}_{\epsilon}$ includes more elements of G as ε increases, and we can expect that actions of elements generating higher equivariance error in general have higher oscillations". (line 274-275)
- line 320-322
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.