Summary
A general methodology is introduced that allows a researcher interested in problems regarding lattices, in the mathematical field of Universal Algebra, to investigate any algebraic property whose validity can be verified on a finite lattice.
The paper indicates how a dataset can automatically be generated . Then a new GNN variant is devised, adapted from previous works (e.g. [1]), that is interpretable. This model is compared to the performance of a vanilla, black-box GNN, with all the other model parameters kept identical.
[1] Steve Azzolin, Antonio Longa, Pietro Barbiero, Pietro Liò, and Andrea Passerini. Global ex-
plainability of gnns via logic combination of learned concepts
Strengths
The paper discusses how a field that has hitherto not seen a large number of AI applications be treated. Two new GNN variants are proposed, called iGNN and HiGNN.
I also like the fact that the well-known lattices N_5 and M_3 where used throughout the text as examples of how the learning algorithms works (Example 3.1 and 3.2).
Weaknesses
__I. Various hyperbolizing adjectives are used throughout the text, that do not provide information:__
- line 18: "Universal Algebra (UA, (6)) is one of the foundational fields of modern mathematics" The importance of Universal Algebra is somewhat overstated. While it indeed is a foundational field, it is usually not part of a typical university curriculum.
- line 43 onwards: "First, we propose a novel algorithm that generates a dataset suitable for training AI models based on an UA equational conjecture. Second, we generate and release the first-ever universal algebra’s dataset compatible with AI [...] Our findings
demonstrate the potential of our methodology and open the doors of universal algebra to AI."
The relevance of the contribution is hyperbolized and overstated, for the following reasons:
1) Indeed, this may be the first application of AI to universal algebra (although experience has shown that in machine learning often new results are introduced, that under a different name were already known previously -- --, so unless an extremely thorough literature review would have been made, I would not feel confident making this assertion).
2) It is known that AI is very well suitable to a particular type of mathematics, where the involved objects are discrete (see [1]), so it is not surprising that this approach works well in this setting too.
3) I am not convinced paper does not "open doors of universal algebra to AI" (see also the claim repeated on line 408), since the method is not tied to the field of Universal Algebra, but rather to the objects of study. Universal Algebra has strong connections to model theory, for example (see [2], chapter 5, which is a reference that is cited in the paper), and exploring these using machine-learning is definitely not covered under the current approach.
line 90: "advanced AI methods" (what exactly makes an AI method "advanced" is debatable)
line 396: "However, as universal algebra is a foundational branch of modern mathematics, any contribution to this field can already have significant implications in various mathematical disciplines."
I feel like these hyped-up claims obscure the actual strong points of the paper: A well-executed experiment with a suitable neural model (GNN) (and other strong points outlined in the Strength section).
__II. Other observations__
- while not a weakness, but a curiosity: it is noteworthy that the lattice examples from Figure 2, page 2, are identical to those from the Wikipedia article on lattices, see Pic 10 and Pic 10 [3]. In Wikipedia, it is mentioned that these are the smallest lattices of their type, but the paper does not mention this (tangential) fact.
- in the appendix, a number of definitions from Universal Algebra are taken from well-knowns, e.g., Definition A.3 corresponds ad literam to Definition 9.3 from [2]. I feel it could be emphasized more strongly that many of the definitions are directly copied from existing sources, rather than merely stating " More formally (6; 8; 16):", where their reference 6 is my reference [2] below.
More precisely, at the beginning of the appendix, it is stated: "An n-ary operation on A is a function that takes n elements of A and returns a single element from the set. More formally (6; 8; 16):" which implies that only the first ensuing definition of arity is taken from references rather than all ensuing definitions.
It would be helpful, to either explicitly state that all theorems/definitions are taken from books (and ideally, like in Theorem A.5, show from where the theorem comes).
- on table from the appendix is missing (below line 582): "Note: ADD TABLE WITH CONCEPT PURITY". This should be fixed, if the paper is accepted, but doesn't make a difference for acceptance/rejection decision.
[1] A.Z. Wagner, Constructions in combinatorics via neural networks, https://arxiv.org/abs/2104.14516
[2] Burris S., Sankappanavar H.P., A Course in Universal Algebra, Springer Millenium Edition
[3] https://en.wikipedia.org/wiki/Lattice_(order)
Questions
- Is there a reason you chose to use linear classifiers instead of decision trees, as the interpretable model of choice?
- consider line 150 ("To address this, we use a hard Gumbel-Softmax activation") vs. line 219 ("The choice of the activation and loss functions to train iGNNs depends on the nature of the task at hand and does not affect their interpretability"). Does that mean that you use different activation functions at different parts of the GNN?
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.