Summary
This paper proposes an extension of symmetrization approach (Yarotsky 2018; Puny et al., 2021; Kaba et al., 2022; Basu et al., 2023) for achieving invariance and equivariance to (small and finite) symmetry groups, with a focus on empirical demonstration of zero- and few-shot transfer learning from non-equivariant pretrained architectures for a range of applications involving different group symmetries. The technical contribution that allows zero- and few-shot transfer is the introduction of some score (rank) function denoted lambda that weights all possible transformed inputs, and using it to turn standard group averaging (Yarotsky 2018) into weighted averaging in a way that the equivariance (and universality) of symmetrization is still guaranteed. The key idea is that an appropriate choice of the score function allows symmetrization to work favorably for the underlying pretrained model (by assigning higher weights to "important" group transformations), which can allow few-shot transfer learning. The authors name this approach lambda-equitune. In particular, choosing the score function as an indicator on argmin of some loss function allows for a canonicalizing symmetrization (Kaba et al,., 2022) that empirically allows zero-shot transfer, which the authors name equizero. The authors experimentally demonstrate the proposed algorithm in a range of applications including reinforcement learning, fairness in language models, compositional generalization, and image recognition under 90-degree rotations and flips.
Yarotsky et al., Universal approximations of invariant maps by neural networks (2018)
Puny et al., Frame Averaging for Invariant and Equivariant Network Design (2021)
Kaba et al., Equivariance with Learned Canonicalization Functions (2022)
Basu et al., Group Equivariant Fine-Tuning of Pretrained Models (2023)
Strengths
S1. The paper aims to address an important and original problem of few-shot or zero-shot transfer of non-equivariant pre-trained deep neural networks to solve equivariant problems. While equivariant transfer learning from non-equivariant model has been investigated by some prior and concurrent work (Basu et al., 2023; Kim et al., 2023), few-shot or zero-shot transfer has not been demonstrated in literature as far as I know. The methodology is clearly motivated and explained, and I think this offers a nice way to steer the behavior of pretrained models towards equivariance with additional controllability offered by the score function or loss function, as demonstrated in the fairness experiment where fairness jointly with high regard score is achieved.
S2. The score function lambda for lamdba-equitune and the loss function for equizero doesn't seem to have to respect group symmetry, which is an advantage as it allows for wider range of choices. While this has been theoretically shown in Kaba et al., 2022, as far as I know this is the first work to empirically utilize the property, since Kaba et al., 2022 proposed but did not experiment with optimization approach.
S3. The applications demonstrated with experiments are quite comprehensive, ranging from reinforcement learning to language generation and visual recognition. Also, the experimental results overall seems to support the main claims of the paper.
Basu et al., Group Equivariant Fine-Tuning of Pretrained Models (2023)
Kim et al., Learning Probabilistic Symmetrization for Architecture Agnostic Equivariance (2023)
Kaba et al., Equivariance with Learned Canonicalization Functions (2022)
Weaknesses
W1. One major ambiguity I find in the paper is that, while the title of the paper and some parts in the main text mention few-shot learning, it seems the experiments concern zero-shot learning or fine-tuning with a fair amount of data. This was confusing to me given that fine-tuning is not equivalent to few-shot learning. Am I missing something?
W2. The approach is only applicable to small, finite groups, due to the requirement of evaluating the score function (lambda) for all possible transformed inputs. This is in contrast to some concurrent work (Kim et al., 2023) that extends to combinatorial or continuous groups, and can be considered a limitation of the proposed algorithm at the current state.
W3. For the fairness in language generation, I think there is a limitation in the proposed algorithm that the considered words (upon which group transformations are defined) are implicitly assumed to be not separated by the tokenizer of the language model. This might not be generally true for modern language models, as a tokenizer can choose to split an equality word into non-equality substrings (a potential example is waitress -> wait + ress). In this case, fairness would not be achieved as expected using vocabulary permutation transformations.
Kim et al., Learning Probabilistic Symmetrization for Architecture Agnostic Equivariance (2023)
Questions
Q1. For the experiments mentioned as few-shot (e.g., Figure 10), how many shots are used?
Q2. In case of zero-shot learning without parameter updates, what kind of practical advantage could we expect from the universality result (Theorem 2)?
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Limitations
The authors have addressed potential limitations and negative societal impact in Section 6.