Not All Neuro-Symbolic Concepts Are Created Equal: Analysis and Mitigation of Reasoning Shortcuts
Neuro-Symbolic (NeSy) predictive models hold the promise of improved compliance with given constraints, systematic generalization, and interpretability, as they allow to infer labels that are consistent with some prior knowledge by reasoning over high-level concepts extracted from sub-symbolic inputs. It was recently shown that NeSy predictors are affected by reasoning shortcuts: they can attain high accuracy but by leveraging concepts with unintended semantics, thus coming short of their promised advantages. Yet, a systematic characterization of reasoning shortcuts and of potential mitigation strategies is missing. This work fills this gap by characterizing them as unintended optima of the learning objective and identifying four key conditions behind their occurrence. Based on this, we derive several natural mitigation strategies, and analyze their efficacy both theoretically and empirically. Our analysis shows reasoning shortcuts are difficult to deal with, casting doubts on the trustworthiness and interpretability of existing NeSy solutions.
Paper
Similar papers
Peer review
Summary
This paper studies reasoning shortcuts (RS) in neuro-symbolic learning. RS refers to that the neural network does not learn a generalizable concept. This work provides a systematic characterization of RS, which not only gives a formal definition of RS, but also identifies key conditions of its occurrence. Several mitigation strategies are empirically or theoretically analyzed.
Strengths
- The paper is well-motivated and well-written. I particularly appreciate the illustrative example in Figure 1. - The definition of RS is derived from causal representation learning, and it is quite reasonable. - Some promising theoretical results are presented, with proofs provided in the appendix.
Weaknesses
- The paper focuses solely on probabilistic logic approaches like DPL, which limits its scope to a relatively narrow scope of neuro-symbolic learning. - This paper introduces too many concepts that may be unnecessary, such as NeSy predictors, unintended semantics, concept extractors, concept distributions, deterministic distributions, etc. Therefore, it may not be easily understood by readers who are unfamiliar with this subfield. - Some more experiments on other baseline methods are needed.
Questions
- The discussed neuro-symbolic learning is essentially in a weakly supervised learning setting (correct me if I'm wrong) [1]. Therefore, can the RS and its analysis be extended to general supervised learning problems? - From my understanding, using Shannon entropy loss is critical to avoid shortcuts. As mentioned in the last paragraph of section 5.4, entropy loss conflicts with NeSy prediction, but [1] proposes using an annealing strategy to gradually remove this entropy regularization. Furthermore, there are quite a few classic methods using maximal entropy-like regularization, e.g., mixup [2], label smoothing [3], focal loss [4], energy-based models [5], and Logit normalization [6]. I strongly recommend separating this part as an additional subsection to reveal this clue for alleviating RS. [1] Qing Li, Siyuan Huang, Yining Hong, Yixin Chen, Ying Nian Wu, Song-Chun Zhu. Closed Loop Neural-Symbolic Learning via Integrating Neural Perception, Grammar Parsing, and Symbolic Reasoning. [1] Zenan Li, Yuan Yao, Taolue Chen, Jingwei Xu, Chun Cao, Xiaoxing Ma, Jian Lu. Softened Symbol Grounding for Neuro-symbolic Systems. [2] Luigi Carratino, Moustapha Cissé, Rodolphe Jenatton, Jean-Philippe Vert. On Mixup Regularization. [3] Rafael Müller, Simon Kornblith, Geoffrey Hinton. When Does Label Smoothing Help? [4] Jishnu Mukhoti, Viveka Kulharia, Amartya Sanyal, Stuart Golodetz, Philip Torr, Puneet Dokania. Calibrating Deep Neural Networks using Focal Loss. [5] Connor Pryor, Charles Dickens, Eriq Augustine, Alon Albalak, William Wang, Lise Getoor. NeuPSL: Neural Probabilistic Soft Logic. [6] Hongxin Wei, Renchunzi Xie, Hao Cheng, Lei Feng, Bo An, Yixuan Li. Mitigating Neural Network Overconfidence with Logit Normalization.
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
No, the limitations of the method were not discussed. The work does not have negative social impacts.
Summary
This paper proposes mitigation and evaluation strategies for "reasoning shortcuts" in neuro-symbolic reasoning models, roughly corresponding to when the concepts (i.e., latent variables) extracted do not match the "true" ground-truth factors. The paper then examines reasoning shortcuts on several proposed, simple datasets, including XOR, MNIST-Addition, MNIST-EvenOdd, MNIST-AddMul, and BDD-OIA.
Strengths
The topic is important and underexplored, while the approach taken in this paper is novel. In general, the presentation is also excellent. The experiments, while fairly simple, are appropriate given the topic. The work is highly related to and extends prior work on disentanglement in machine learning, which is referenced (though not especially explicitly discussed). It is also decently well-motivated and some of the tasks like the BDD dataset highlighted interesting failure cases.
Weaknesses
Fundamentally, this paper is subject to many of the same challenges as prior work on disentanglement. That is, the task of identifying the "correct" latent variables is the task of disentanglement. While Locatello et al. (2019)'s "Challenging common assumptions" is cited for the definition of disentanglement (as [44]), the key lesson is never explicitly referenced (at least, not attributed): without supervision or inductive biases on the model and data, disentanglement is impossible. However, the authors highlight a similar point empirically, that "Disentanglement is not enough under selection bias." Of course, this is not a particularly high bar, and indeed, the strategies taken to mitigate reasoning shortcuts in this paper do so with varying degrees of explicitness. For example, the paper's "data-based mitigation" explicitly provides supervision using a subset of the latent variables. However, in their objective-based approach, which draws inspiration from autoencoders, they could have plausibly also built on disentanglement-focused literature like InfoGAN (Chen et al. 2016) or $\beta$-TCVAE (Chen et al. 2018). However, with that being said, it approaches many of these questions from a refreshing new perspective. This leads to the second concern - to some extent, many of the proposed mitigation strategies are standard in the disentanglement literature. For example, "On the relationship between disentanglement and multi-task learning" (Maziarka et al. 2021) highlights the role of training on multiple tasks for learning disentangled representations -- in this paper, that is seemingly equivalent to knowledge-based mitigation. On the other hand, the data-based mitigation seemingly corresponds to the approach taken in "Semi-Supervised StyleGAN for Disentanglement Learning" (Nie et al. 2020). The objective-based mitigation (i.e., reconstruction loss) is, of course, central to many representation learning approaches. Then in the discussion of "architecture-based mitigation," the connection to disentanglement work is made explicit. However, approaching these questions from the perspective of reasoning is fairly novel and is an important connection. Ultimately, while the problem itself is well-motivated and novel, and the paper itself is excellent, and I believe it should likely be accepted based on the analysis alone, my primary reservation stems from the question of how reasoning, in particular, has shaped the mitigation strategies taken in this paper, beyond the datasets considered. ---- Post-rebuttal update ---- Most of my concerns have been addressed - I've raised my score to a 7.
Questions
What differences would you highlight between the reasoning-focused tasks in this paper and the more traditional tasks in the disentanglement literature? How are the approaches discussed in this paper particularly well-suited for those tasks?
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
4 excellent
Presentation
4 excellent
Contribution
3 good
Limitations
I think some more explicit discussion of the theoretical limitations demonstrated by related works (e.g., Locatello et al. 2019) would have been useful.
Summary
_Background_: The authors are interested in studying the properties of sequential, two-stage neuro-symbolic pipelines. Stage 1 is a neural network that reduces a high dimensional input (eg: an MNIST image) to a low-dimensional relaxation of a symbolic space (eg: 10 dimensional onehot vector) and Stage 2 is a probabilistic function (neural network or otherwise) that predicts the labels given the symbols utilizing a provided knowledge base. However, when trained end-to-end, the symbolic representation often collapses and the symbols learned are not equivalent to the ground truth symbols. This is called a reasoning shortcut (RS). This work focuses on investigating the causes of such reasoning shortcuts and presenting suitable remedies for the same. The authors formally define reasoning shortcuts for NeSy architectures as symbolic mappings which allow the model to achieve a locally optimal loss, but whose semantics do not match the ground truth semantics. The authors identify four factors to mitigate the occurrence of reasoning shortcuts: 1) additional prior knowledge, 2) additional data, 3) additional reconstruction targets in the objective function, and 4) additional inductive bias in model architecture design. They study the impact of these reasoning shortcuts mitigations on five NeSy prediction tasks with three NeSy algorithms. The authors find that different combinations of mitigation strategies helps improve performance in different cases.
Strengths
**Originality:** - While the concept of reasoning shortcuts was previously known, this paper is the first to offer a formal definition and analysis of reasoning shortcuts as a limitation of instantiations of neurosymbolic models. - The mitigation strategies proposed are widely used in the neurosymbolic community, but they haven’t been connected before under the umbrella definition of reasoning shortcut mitigations. Hence, this paper brings a fresh perspective to why such strategies work well for NeSy systems. **Quality and Clarity** - The setup, definition, and analysis of reasoning shortcuts is well motivated and well explained. **Significance** - The paper identifies a significant problem with previous work and provides a formal definition of the problem and mitigation strategies to address the problem. - I’m intrigued by the analysis of counting reasoning shortcuts because it presents a method to quantify how well a knowledge base describes a dataset / data generation process.
Weaknesses
**Clarity:** - Table 1: DIS is mentioned in the caption, but no row for it exists in the table. - Table 2: It took me a long time to figure out that each number is the frequency of RSs for a baseline on a dataset. Please explain what the rows, columns, and values mean for each table. - Line 276: “when forcing disentanglement…” Disentanglement is a property of the architecture. How is the architecture modified to enforce disentanglement? The appendix doesn't mention this. - “Roughly speaking, a concept F1 below 95% typically indicates an RS” I’ll defer to the author's judgement here because I’m not familiar with the data, but to me, a low concept F1 *on its own* signifies that the symbol extraction hasn’t been successful. Did the authors perhaps mean a low concept F1 but a high prediction F1 hints at the existence of RS's? - Table 4 results: I'm not sure that I understand the results here. I defer to the authors, but I think this table and task demands extra scrutiny. - The prediction accuracy and the concept accuracy both seem low (< 80% as a ballpark figure). Is this close to the state-of-the-art performance for this dataset? - "concepts tend to align much closer to the diagonal..." I don't quite understand how being closer to the diagonal implies better concept generalization. The concepts aren't ordered, so an incorrect prediction, regardless of perceived distance from the correct concept, should be incorrect. Maybe I'm misunderstanding this sentence. **Significance:** - The method only compares against ablations of the same model. I'm concerned that this paints an incomplete picture about the relative performance of NeSy predictors compared to other models on these tasks. This lowers the significance of this architecture for the neurosymbolic community. I'd advise the authors to look into other fully neural and neurosymbolic methods that might have similar problem statements, and/or provide reasons for why the baselines are not suitable for the datasets. I've provided a couple of works I'm familiar with that are related (but not exactly equivalent) to the base algorithm studied in this paper. * ROAP[1]: ROAP presents an end to end algorithm for symbol grounding and symbol manipulation when external knowledge is not available, but we have diversity of tasks to instantiate a multi-task learning objective. I am reminded of ROAP because one of their tasks is similar to the MNIST digit grounding tasks presented here. * Concept Embedding Bottleneck models[2]: Concept embedding bottleneck models instantiate an extra layer that creates a concept embedding that's used for downstream prediction. I am reminded of CEM models because their algorithm is geared towards leveraging explainable symbolic information in real world tasks without loosing accuracy. * I'd also be interested in whether the authors think these models use the mitigation strategies presented for reasoning shortcuts. Overall, I'm giving this paper a __weak accept__. The authors formalize an interesting problem which is definitely interesting to the broader neurosymbolic community. However, the related work / evaluation focuses on a very specific definition of neurosymbolic models. [1] https://arxiv.org/abs/2206.05922 [2] https://arxiv.org/abs/2209.09056
Questions
See weaknesses section
Rating
6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
Confidence
2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness
3 good
Presentation
3 good
Contribution
3 good
Limitations
The authors have adequately addressed potential limitations.
Summary
The paper presents a in depth analysis of reasoning shortcuts and the impact they have of the learning process of neuro-symbolic learners. On the ground of the performed analysis, they propose 4 different mitigation strategies to alleviate the problem: 1. Knowledge-based mitigation 2. Data-based mitigation 3. Objective-based mitigation 4. Architecture based mitigation
Strengths
The paper studies an important problem and it is very well presented. It is easy to follow and the formalisation of the concept of reasoning shortcuts was needed. The experimental analysis supports the claims.
Weaknesses
I have only couple of perplexities regarding the first two mitigation strategies: 1. in the knowledge-based mitigation, the authors claim that depending on the application, collecting new knowledge might not be feasible. They then proceed in proposing Multi-task learning as a practical alternative. Here it is were I get confused: - how can this be more practical? In addition to collecting more knowledge you now need to collect additional labels. - if I understand correctly, in proposition 4 in the end the conclusion is anyway that the deterministic optimum $p_{\theta}(\mathbf{C} \mid \mathbf{G})$ of the MTL loss is a deterministic optimum of a single task whose prior knowledge is the conjunction of all the prior knowledge used in the different MTL tasks. How is it then better defining different MTL tasks than collecting more knowledge? 2. In section 5.2 the authors say that RSs can occur also when the dataset is exhaustive. This to me seems more of a problem related to not choosing the right concepts, for the task, i.e., there is not a bijective correspondence between the concepts in $\mathbf{C}$ and $\mathbf{G}$. For example, suppose we have the (very simple) task of distinguishing bird images from non-bird images, and suppose we have one concept encoding the classes "pigeon, flamingo, duck" (and we have the simple background knowledge that any of these classes implies "bird"). In this case, one mitigation strategy could be to add more supervisions on the concept (as proposed), but wouldn't it be better to actually take concepts that are "meaningful" for the task? On the line of the last point above, I was very surprised from the amount of reasoning shortcuts learnt in the MNIST-addition task in table 2. Can you provide some examples for such RSs? Also, these results are taken on *optimal* runs. What would be the results on 30 *standard runs*? Also, can you provide some results in terms of accuracy on both the concepts and the final outcome when using disentanglement and when not?
Questions
See above
Rating
7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness
3 good
Presentation
4 excellent
Contribution
3 good
Limitations
Not applicable
Thanks for the response! > **Disentanglement vs. RSs** > As for specific differences, notice that research on disentanglement focuses on continuous latent factors (e.g., in non-linear independent component analysis and disentangled representation learning via VAEs), does not consider symbolic knowledge, and typically allows factors to be permuted. In contrast, in our work, we deal with discrete concepts learned from prior knowledge and do not allow permutations of factors. I do not believe this fully matches the use of disentanglement in the literature. While discrete, ordered variables have not been the primary focus of disentanglement literature (arguably, out of convenience), both aspects have been explored. The question of how to learn discrete latent factors has attracted a decent amount of interest in several contexts, and one can find dozens of papers on this topic [1-4, and many others]. Moreover, most (probably all standard) disentanglement methods that provide supervision do not allow arbitrary permutation of variables (e.g. [5] mentioned in the original review and 6]). > **Strategies are not novel** As noted in the original review, I believe the framing and specific motivation in this work are novel, and in general, am positive about this work. However, given that there is a body of prior work which aims to implement similar techniques to these mitigation strategies, I believe it is worth highlighting these connections more explicitly. Refs: 1. Learning Disentangled Joint Continuous and Discrete Representations, Dupont 2018 2. Disentangling generative factors in natural language with discrete variational autoencoders, Mercatali and Freitas 2021 3. Structured Disentangled Representations, Esmaeili et al 2019 4. Learning Disentangled Discrete Representations, Friede et al 2023 5. Semi-Supervised StyleGAN for Disentanglement Learning, Nie et al 2020 6. Weakly-Supervised Disentanglement Without Compromises, Locatello et al 2020
Reply by Authors
Thank you for the quick reply and your effort in engaging with us! In light of this discussion, **we agree related work on disentanglement should be covered more thoroughly in the main text, and we plan to do so in the Related Work section**. Please let us know if you’d prefer we address this differently. **On disentanglement.** We stand corrected, you are right that there exists also work on disentanglement of discrete variables. We acknowledge that there exist different definitions of disentanglement in the literature. The reason why we distinguish between disentanglement and identifiability is based on our reading of [Suter et al., Reddy et al., Hyvarinen et al.]. Our view is that factors are disentangled iff they can be manipulated (through do-operations) independently from each other [Suter et al., Reddy et al.], while identifiability amounts to finding the "correct factors’’ [Hyvarinen et al.], regardless of whether are disentangled or not. There are also weaker notions of identifiability (such as "weak identifiability’’ [Hyvarinen and Morioka] and "alignment’’ [Marconato et al.]) that lie in-between disentanglement and identifiability. Note also that if the ground-truth factors are correlated to one another, and the learned concepts correctly identify them, then these cannot be disentangled (according to [Suter et al., Reddy et al.]). Our analysis also covers this case, in that it makes no assumptions on the G’s being disentangled. This is why we are keen on keeping this distinction. [Suter et al.] Suter, Miladinovic, Schoelkopf, Bauer. “Robustly disentangled causal mechanisms: Validating deep representations for interventional robustness.” ICML 2019. [Reddy et al.] Reddy, Balasubramanian, Vineeth, and others. “On causally disentangled representations”. AAAI 2022. [Hyvarinen and Morioka]. "Nonlinear ICA of temporally dependent stationary sources." Artificial Intelligence and Statistics. PMLR, 2017. [Hyvärinen et al.] "Identifiability of latent-variable and structural-equation models: from linear to nonlinear." arXiv:2302.02672 (2023). [Marconato et al.] “Glancenets: Interpretable, Leak-proof, Concept-based Models”, NeurIPS 2022. **Additional pointers to strategies for disentanglement.** We now see better how to address this: we plan to cite relevant prior work on approaches in line with the mitigation strategies we identified in the main text. We never intended to dismiss these works.
Thanks for the clarifications! I'm keeping the score where it is right now.
Reply
Thank you for clarifying my concerns. I would like to further discuss the Shannon entropy regularization. I agree that using a Shannon entropy with a constant coefficient may cause conflicts. However, this issue can be properly addressed by applying an adaptive strategy, such as focal loss [4] or annealing strategy [1]. In general, this is a good paper, I have raised my score to 7.
Decision
Accept (poster)