MatrixNet: Learning over symmetry groups using learned group representations

Summary

In this paper, authors study the question of what feature representations to use for learning tasks with inputs coming from a symmetry group. They propose MatrixNet, a neural network architecture that learns matrix representations of group element inputs instead of using predefined representations. The main contributions are as follows: 1. Formulation of the problem of learning over groups as a sequence learning problem in terms of group generators with invariance to group axioms and relations. 2. The proposed neural network architecture, MatrixNet, achieves higher sample efficiency and generalization over several more general baselines in prediction tasks over the symmetric group and Artin braid group. 3. The matrix block method for constraining the network to respect the group axioms and an additional loss term for learning group relations.

Strengths

The strength of the paper is building up links among group reprsentation theory, learning tasks and neural networks, more specifically: 1. Formulation of the problem of learning over groups as a sequence learning problem in terms of group generators with invariance to group axioms and relations. 2. The matrix block method for constraining the network to respect the group axioms and an additional loss term for learning group relations.

Weaknesses

The motivation is not so clear to me. The applications of group theory are everywhere in the real-world. Thus, how to choose a good representation for the group which is associated with the learning task is more important. In this paper, authors formulate this problem but do not give a clear answer (see my questions below), and authors focus on solving a more abstract group theory question instead which is already studied a lot by mathematician. I didn't see any insight behind the paper. For the experiment, I think it only solved a mathematical problem at simple setting which is far from what we expect. For example, how the model works when the group's order become larger?

Questions

1. The motivation is to solve a mathematical problem or real-world learning task? I hope it can help me understand the paper clearly and inprove the writing. 2. In subsection 4.1, authors formulate the problem. In line 166, can $f$ be seen as a representation of $G$ or not? If it is a representation, can you write a explicite expression as an example? Since if it is a representation, the target domain $R^{c}$ may not be a linear space which makes it unclear for me. 3. Can you show more experiment results? Such as larger symmetric group? Also I didn't understand the meaning of predict the order of the group element? I think we should determine it instead of estimating it? Maybe I did't get your point, but I am open to discuss.

Rating

5

Confidence

4

Soundness

2

Presentation

2

Contribution

2

Limitations

see the weaknesses and questions.

Summary

The paper describes MatrixNet, a method to learn group representations such that they are optimized for a certain task of interest using a neural network. , The neural network takes in a group element in the form of a sequence of generators that compose to form the group element and forms an intermediate output that contain the matrices that are the group representations for each of the generators. The generators can them be multiplied to form the group representation which is then fed to further neural layers that are task-specific. The network is trained so as to (approximately) satisfy the constraints of group representations using architectural constraints and the loss function. Experiments on predicting the order of a element of the symmetric group and Jordan-Holder multiplicities for the 3-strand braid group.

Strengths

1. The paper present new ideas for learning on groups. Design of architectures is clever and done well with good mathematical justification. The architectures provably output group representations. 2. Experiments show that the proposed architecture leads to learning group representations that lead to better results compared to just using 3. The paper also does a good study of the possible variations of MatrixNet and presents experimental results for them.

Weaknesses

1. One of the weaknesses is that the parts about the braid group are very difficult to follow and probably need a lot more background for readers and attendees for this conference. This includes myself and I was not able to fully follow the details or why that experiment is important. 2. The experiments are also a little weak in my opinion. It was not clear to me how the learned representations were different from precomputed ones and why they were better for a given task. Also, what happens when the size of the group representations is larger. Perhaps for the first experiment, it would be nice to see the results as a function of the size of the group representation. I am not sure if the proposed architecture and learning mechanism can learn good representations when the size is large. This is important to address, in my opinion.

Questions

No additional questions.

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

Yes, authors have listed the limitations of the current submission and suggested directions for future research.

Summary

This paper studies feature representations of a group element for supervised learning. It considers a regression task where an input is a group element g of a finite group and a target is some label. Firstly, g is decomposed into a sequence of generators (g_1, ..., g_n) such that g = g_1 \circ ... \circ g_n. Next, each generator is mapped by a trainable matrix embedding W: Gen(G) --> R^{n \times n} followed by the matrix exponential so that we can get a group representation M_i = exp(W(g_i)) ∈ GL(n). Then their product M_1 \circ ... \circ M_n becomes the feature. Similar variants are also proposed. The performance is evaluated on two synthetic tasks: the order prediction of the symmetric group S_10 and the action prediction of the Braid group B_3.

Strengths

1. The paper is well written. Technical details are clear enough. 1. The idea to construct a feature representation of a group element as a learnable parameter is interesting and it is a possibly promising direction.

Weaknesses

1. Limited applicability to real tasks. To the best of my knowledge, regression (or classification) of group elements is practically important for continuous groups such as SO(3) (e.g., pose estimation). However, the current approach is only applicable to finite groups. Also, I think the tasks conducted in the experiments are not directly bridged to real problems, and I feel uncertain about how the proposed method is practically valuable. 1. It is unclear which component significantly contributed to the final performance gain. The proposed method consists of (at least) two parts: the decomposition of g into generators and a trainable representation of a generator. The current experiments are not designed to evaluate them separately. 1. The experiments are not convincing enough—they are relatively small scale, use synthetic tasks only, and have less variety (two tasks).

Questions

1. Given g, is its decomposition into the generators always uniquely determined? 1. In the experiment of 5.1, what will happen when we employ the group decomposition while using the fixed representation? I mean, the feature of g is given as concat[\rho(g_1), ..., \rho(g_n)] where g = g_1 \circ ... \circ g_n and \rho(g_i) is some representation of g_i (e.g. irreducible rep). Typo: * One of M_{g_ik^-1} would be {M_{g_ik}}^{-1} in the equation below line 203.

Rating

4

Confidence

3

Soundness

3

Presentation

3

Contribution

2

Limitations

Limitations are adressed.

Summary

The authors use neural networks to learn group representations. A group element is represented by its generators formatted as a sequence of learned matrix representations. These generators are mapped to a single matrix representation of the group element via the Matrix Block which enforces group axioms. The resulting feature is used for downstream tasks on groups. The authors show that the proposed architecture MatrixNet successfully predicts group element orders for S_10 and Jordan-Holder multiplicities for braid group B_3. Further analysis on word length extrapolation and visualization demonstrates the superiority and usefulness of the approach.

Strengths

1. In Table 1, the faster convergence of MatrixNet potentially indicates that the proper architectural constraints provide good inductive bias for learning group representations and solving downstream tasks over groups. It’s good to see how one can explicitly build in these constraints in the problem of group representations and that it works better than a less constraining MLP without domain-specific inductive bias. 2. The paper is well-written.

Weaknesses

1. It’s unclear why naive MatrixNet and MatrixNet-MC cannot extrapolate to longer word length. 2. Empirical evaluations are a bit limited. It might be helpful to evaluate groups with different properties (see questions).

Questions

In the matrix block, is it possible to introduce commutative operations between different matrix representations of group generators if the input group element comes from an abelian group. It seems possible to introduce further architectural constraints for specific groups.

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

None

Thank you for the response

Thank you for answering my questions. I believe the authors have done a good job in addressing most of the concerns that all the reviewers had. The authors should definitely include all the new results in the rebuttal into the main paper. I will raise my score to a 7.

To authors

Thank you for your response. My concerns are almost addressed. Still, I keep the original score because I'm not convinced enough of its applicability. Since I'm not a mathematician and I cannot judge how the group problems are significant.