Neuro-symbolic Learning Yielding Logical Constraints

Summary

This paper takes inspiration from convex and bilevel optimization to derive a learning algorithm that simultaneously learns neural network perception and rules on the perceptions. The authors derive an efficient training algorithm involving multiple minimization steps. Post rebuttal: I thank the authors for addressing the concerns. I will keep my score, this is a good paper, and I am happy error bars are included in the rebuttal.

Strengths

The paper tackles the genuinely very hard problem of learning rules and perception simultaneously. The performance on, from what I can tell, quite challenging tasks is great. Ablations show multiple complex techniques help with performance. Furthermore, the paper has several proofs on convergence and optimality. The method itself has many (original) interesting ideas, although there are many.

Weaknesses

While the paper is decently written, there are a lot of technicalities to the method that makes understanding the high-level picture challenging. It relies on many optimization methods: Cardinality constraints, difference of convex programming, proximal point algorithms and trust regions, not to mention quite a few hyperparameters. Some equations fly in without a clear justification, and glancing at the appendix does not help me either. Furthermore, the paper introduces a hyperparameter alpha that significantly complicates the computation without actually using it. The main goal of this paper is to yield logical constraints, but whether they are also interpretable is not evaluated. The paper does not have error bars on experiments.

Questions

- An SMT solver is mentioned multiple times of which in Figure 1. Is my understanding correct that it is only used during inference, not training? - Would the method also work by taking the cross entropy for the $\ell_1$ loss? - Page 4, 131: Why is it allowed to rewrite the objective introducing alpha like that? - Proposition 1: Explain what $\mathbf{e}$ is. - Page 5, 154: The linearization uses $(e-2u)^Tu$, but further down (line 172) it is $(e-w)^Tw$. Where did the 2 go to? - Same question for neural network training. Also, I think there's something wrong with the braces there. - To what values do you set $m$ (number of rules)? - Appendix, table 1: You mention that $\alpha$ is fixed to 1, meaning the influence from the rules is ignored by symbol grounding. Why did you choose this setting? $\alpha$ significantly complicates your mathematics. Setting it to 1 would simplify the equations significantly.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

2 fair

Contribution

4 excellent

Limitations

Inference of this method is very expensive, as highlighted in the limitations section. I'd have preferred this in the main paper, not in the appendix.

Summary

This work provides a neural framework that combines network training, symbol grounding, and logical constraint synthesis. It utilizes the cardinality constraints to express the logical constraint learning and a DC penalty for constraint relaxation. The evaluation demonstrates that this method outperforms state-of-the-art models including both SATNET and L1R32H4 by a large margin.

Strengths

Originality: 4/5 Pros: This methodology introduces two novel important components, DC relaxation loss, and cardinality constraints, into the numerical learning framework. Cons: One interesting component in this paper, mapping the learned numerical rules into the symbolic space is not quite clearly explained. By studying the related work section, I think the "Softened Symbol Grounding for Neuro-symbolic Systems"[1] seems quite related to the rule extraction part. I feel this work is probably not emphasized enough in the paper's related work section. Quality: 4/5 Pro: The theory and properties come with all the proofs explained in the appendix, which is quite convincing. This work also reaches better than the SOTA performance for the two experiments. Cons: The SATNET* uses Distilled LeNet as its underlying perceptual model, while you have used a recurrent transformer model. This seems an unfair comparison. Clarity: 3/5 Pros: The math component is nicely defined and with well-explained definitions. Cons: It is hard to understand the process of extracting a constraint from its numerical form ($\mathbf{w}$ and $\mathbf{b}$). Significance: 4/5 Pros: This work improves from the existing work by learning constraints along with ground truth perception, which is an important task in neural symbolic learning. Cons: The learned constraints are in boolean form and thus not high level enough to generalize across different task variants. [1] Li, Zenan, et al. "Softened Symbol Grounding for Neuro-symbolic Systems." The Eleventh International Conference on Learning Representations. 2022.

Weaknesses

See Strength. Minor comments: 1. Line 91: There are two papers coming from different groups, whose first author's names are both Li. It seems that the two works come from the same person in the article. 2. I would recommend putting the limitation section into the main paper.

Questions

1. What is the limitation of using cardinality constraint? Is there a fundamental limitation other than the tricky usage where introducing auxiliary variables is required? 2. How are the 324 cardinality constraints calculated? 3. Suppose the number of the constraint is unknown for a new task, what should a user do? How will that impact the performance? 4. How to interpret a boolean-based constraint to an average programmer? I am happy to raise my score if these questions are addressed.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

Suggestions included in the Question section.

Summary

The authors propose a fusion of neural network and symbolic domain via logical constraints to learn specific vision tasks in a weakly supervised way. They break it down to two optimization problems for neural and symbolic domains. The logical constraints are solved using a deterministic solver and grounded softly to the neural component via a latent space $z$.

Strengths

* The neuro-symbolic approach of solving tasks that require some planning is always an interesting direction. * The results on the two domains studied in the paper are quite strong, indicating that such fusion can indeed converge well

Weaknesses

* The paper needs more high level figures to help understand the exact logical constraints being enforced for both the domains. This will significantly improve the readability. * Please add some discussion about the limitations of using the specific cardinality-based logical constraints you chose in terms of where it might be applicable vs where it cannot.

Questions

* Why do you chose these specific tasks? Is it following some prior works? Please make it clear. This will help understand the scope of the work in terms of applicability to different domains.

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

1: Your assessment is an educated guess. The submission is not in your area or the submission was difficult to understand. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

* The authors don't have any limitations section. Please refer to weakness for recommendations on this.

Summary

The paper proposes a new neurosymbolic approach for learning symbolic representations and logical constraints on top of these simultaneously. The authors propose a new penalty term based on Difference of Convex (DC) programming in order to relax the optimzation problem. The performance of the proposed approach is demonstrated on the visual sudoku challenge and a path planning environment based on the Kitti and NuScene datasets. Overall, the method performs better than existing works and achieves results close to a fully supervised architecture in some cases.

Strengths

The paper takes a principled approach to the problem of learning symbolic representations and logic rules simultaneously. The proposed DC programming approach allows for the theoretical analysis as per Theorem 1 demonstrating that gradual increase of the weight of the DC penalty term results in the desired stationary point. Also, the reported results demonstrate strong improvement over other neurosymbolic approaches in the two considered environments of the visual sudoku and path planning.

Weaknesses

The performed experiments focus on 2 relatively simple and static environments. Moreover, one of the core motivations behind neurosymbolic approaches is the ability to learn faster and/or transfer knowledge to new environments, however none of these questions have been addressed by the conducted experiments. Overall, I think the paper would benefit from making the methodology section more concise and to the point and expanding more on the results.

Questions

- What does the notation (z;y) mean exactly in w^T(z;y)? Is it some form of concatenation? - You mention that the solvers can find a solution even with incorrect perception. Why do you think this happens? Do you think that using more powerful solvers can actually be a problem for the perception side of things?

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

2 fair

Contribution

3 good

Limitations

The paper does not discuss the limitations of the proposed methodology which is important in order to identify meaningful directions for future work.