Interpret Your Decision: Logical Reasoning Regularization for Generalization in Visual Classification

Summary

The paper introduces a logical regularization method called L-Reg, aimed at enhancing the generalization ability of image classification tasks through a logical reasoning framework. L-Reg effectively simplifies the complexity of the model by ensuring that the generated atomic formulas align with the logical relationships between the images and labels, promoting balanced distribution in the feature space, and reducing the number of extreme weight values in the classifier. Both theoretical analysis and experimental results validate the effectiveness of L-Reg in various generalization scenarios, especially demonstrating outstanding performance in multi-domain generalization and generalized category discovery tasks.

Strengths

1. L-Reg effectively reduces the complexity of the model by balancing the feature distribution and reducing the number of extreme weight values in the classifier. 2. In Sections 3.1 and 3.2, the paper provides logic-based theoretical analysis and detailed derivation of the construction process of L-Reg. Furthermore, through experiments in Sections 4 and 5, the effectiveness of L-Reg in different generalization settings, especially in multi-domain generalization and generalized category discovery tasks, is validated. 3. Designed as a plug-in loss function, L-Reg is compatible with most existing frameworks, making it highly flexible and practical in real-world applications. 4. The paper also explores the relationship between logical reasoning and visual classification tasks, delving into the derivation of logic-based regularization terms to promote generalization, providing new perspectives and methods for research in related fields.

Weaknesses

1. L-Reg may reduce the scope of semantic support, leading to a slight performance decrease on known datasets. It is hoped that the authors can analyze in more detail the reasons for this performance drop and provide possible improvement methods. 2. The appropriate function g for generating atomic formulas is a key factor. While the authors propose L-Reg as a regularization method to ensure that F(g(Xs), Ys) consists of atomic formulas, they do not elaborate on how to choose or design this function g. Further explanation on the specific methods or criteria for selecting g is desired. 3. In multi-domain generalization tasks, how L-Reg integrates with existing methods and how to adjust alpha to balance the two losses are details that the authors are encouraged to further elucidate.

Questions

I hope the authors can further compare L-Reg with other regularization methods in terms of generalization ability and interpretability to highlight the advantages of L-Reg.

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

Authors have adequately addressed the limitations.

Summary

This paper addresses the multi-domain generalization (mDG), generalized category discovery (GCD), and the more challenging mDG+GCD task. The authors introduce a logical reasoning-based regularization term called L-Reg, which bridges logical analysis with image classification to enhance model interpretability and reduce complexity. The main idea of L-Reg is to identify a minimal set of semantics that can deduce the relationship between $x$ and $y$, i.e., the semantic support. Theoretical analysis and experiments demonstrate that L-Reg improves generalization across mDG, GCD, and mDG+GCD.

Strengths

1. This paper establishes a connection between logical reasoning and practical visual generalization problems, bringing novel insights for improving DG and GCD. 2. The proposed L-Reg used over existing SOTA methods can further improve the SOTA performance. 3. The experiments are comprehensive. The visualization results provide a good empirical understanding of the role of L-Reg.

Weaknesses

1. The presentation of this article is somewhat confusing for readers unfamiliar with logical reasoning, especially from Section 3.1 to Section 3.2. It would be better to add intuitive explanations of key concepts. For example, why should $F$ be atomic formulas for a logic to be a 'good general' one, or, what do the atomic formulas refer to in a real-world case? 2. The mDG improvements of L-Reg are over GMDG. The performance of directly applying L-Reg is not shown, i.e., ERM+L-Reg.

Questions

1. Line 145 claims that "Semantics that occur frequently across samples often lack decisiveness for classification." However, samples from the same class often share the semantic predictive of the class. 2. The definition of the "minimal semantics" $z_i$ is vague. The meaning of $Z_i$ is also unclear in Eq. (3). 3. Could you provide a concrete implementation of L-Reg? I wonder which layer(s) is(are) chosen for computing Eq. (3) in practice? Are the results sensitive to the selection of layers?

Rating

6

Confidence

2

Soundness

3

Presentation

2

Contribution

3

Limitations

Yes.

Summary

This work introduces a sample-based regularization technique, L-Reg, which goes beyond techniques like parameter-based L2 regularization by being more interpretable and demonstrating better generalization ability. The work formalizes the notion of semantic support to force the model to learn minimal sufficient statistics, quantitatively and qualitatively showing how that it leads to better generalization across multiple settings - multi-domain generalization, generalized category discovery and a new setting that is a combination of the two which they introduce.

Strengths

+ A theoretically grounded paper with comprehensive experiments and results. + The paper is well written in general. + I especially liked how the algebraic logic formalism was neatly tied into this space. The idea of using semantic supports, although simple, is motivated and formalized well. I also liked how the negation of the semantic support set was used to formulate the optimization problem. + The derivation of conditions required to hold under various settings is well done and makes the derivation of the regularization easy to follow. The proposed mDG + GCD setting is interesting.

Weaknesses

* Although the method introduced is technically sound, with the baselines being quite comprehensive, the improvement in results seems minor. This suggests that accuracy may not be the right metric to compare against here. Considering the objectives proposed, shouldn’t other metrics beyond accuracy be considered? * The paper states that “the semantics generated by the encoder and classifier can be combined to form atomic formulas”. I expected to see some of the actual learnt atomic formulae in the results - which I did not. * From the formulation of the L_reg loss, it appears that any concept-based model that is sparse may be similar to the proposed formulation. How is the proposed method different from such methods -- formally and empirically?

Questions

* Since the paper mentions “constructing atomic formulae”, would it be possible to actually extract and see how these look from the model? This might significantly strengthen the paper. * How does this method compare, formally and empirically, with sparse concept-based models? Wouldn’t their loss be very similar to the proposed L_reg loss?

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

Yes, limitations have been addressed.

Summary

The paper proposes a novel logical regularization termed L-Reg for visual classification. L-Reg encourages models to focus on the salient semantics and thereby emerges interpretability. The theoretical analysis provides clear connections between logical reasoning and L-Reg. Extensive experiments demonstrate that L-Reg also benefits the generalization of models to unseen domains and categories.

Strengths

1. Studies on loss regularization have positive influences on various fields. 2. The paper is well-presented, and L-Reg is clearly presented with rigorous theoretical analysis. 3. The qualitative benefits of L-Reg are validated through experiments, and the generalization brought by L-Reg has been demonstrated sufficiently under three settings.

Weaknesses

1. As stated in the introduction section, interpretability is a longstanding focus among studies in regularization terms. The authors claim that L2 regularization might lead to ambiguous interpretability, which also serves as an important motivation and contribution for L-Reg. However, the interpretability of L-Reg has not been adequately discussed except for the introduction section. Further analysis from either qualitative or quantitative perspectives can greatly strengthen the presentation. 2. The analysis of Figure 5 should be detailed, especially the examples of unknown classes in Row 3.

Questions

See weakness.

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

The author have discussed both the limitations and potential societal impact. The limitations are left for future research.

Summary

This paper mainly focuses on two problems: 1) How does logical reasoning relate to visual tasks such as image classification? 2) How can we derive a logical reasoning-based regularization term to benefit generalization?. Then, this paper proposes a method called Logical Reasoning Regularization based on the analysis of the two problems. Theoretical analysis and experimental results demonstrate that L-Reg enhances generalization across several scenarios.

Strengths

1. The main contributions of this article are: 1) Building the relationship between logical reasoning and visual tasks such as image classification; 2) Rethinking the classification task from the logical reasoning perspective and proposing Logical Reasoning Regularization. Overall, the contributions are meaningful, and the paper is interesting. 2. The paper is easy to read.

Weaknesses

1. As can be seen from Table 1 and Table 2, the proposed regular term has a weak improvement on the existing method. 2. The analysis of this paper is incomplete and lacks theoretical analysis of sufficient conditions or sufficient and necessary conditions for meeting Atomic Formulas. 3. Overclaim. This paper claims that the proposed L-Reg can reduce the complexity reduction. However, the proposed L-Reg is regarded as a regular term and is directly added in the learning objective. This does not reduce the computational complexity of the training period, but the extra regular term increases the computational overhead of the whole training process.

Questions

1. line 123, f_s should be f. 2. In Figure 1, it can be seen from the visualization results of the first line that the learned features are basically concentrated on the background, and it is felt that the obtained model is overfitting or underfitting. I would like to know the specific parameter Settings, experimental codes, random seeds, etc., of the experiment. 3. In Figure 3, what does the abscissa represent? What is the baseline being compared? To make the results more convincing, an additional baseline and dataset need to be added. 4. In Figure 4, the coordinate proportions of the left subgraph and the right subgraph are inconsistent, so it is unfair to make a direct comparison. Secondly, the horizontal and vertical coordinates need to be explained what they represent. Finally, the left and right subgraphs are almost identical, It cannot be proved that +L-Reg can achieve real elimination of certain extracted semantics characterized by dominant frequencies across all samples. 5. In lines 173-186, Why the constraint in eq. (6) is removed in eq. (8), please give a detailed derivation process.

Rating

6

Confidence

3

Soundness

2

Presentation

3

Contribution

2

Limitations

The authors have adequately addressed the limitations.

Thanks for response

Thank you for your kind feedback. We will ensure that the relevant discussions are thoroughly incorporated into the final manuscript as suggested.

Response to author rebuttal

I thank the authors for the detailed rebuttal, and the efforts. * The additional results, esp the comparison with sparse concept models, is useful. I'd suggest that this should be included in the main result tables. Since one of the significant claims of this work is the reduction in complexity of parameters, comparisons with sparse models would be necessary to show the usefulness of this approach. * Thank you for the qualitative example and explanation on the atomic formulas. It would be great to include a few qualitative results (positive and perhaps even cases where the method failed) in the appendix of the paper. This would greatly help understand the paper better. The paper is meritorious, and I stay with my rating of WA.

Gratitude to All Reviewers

We would like to express our sincere gratitude to all of you for your valuable feedback and participation in the discussion regarding our paper. **We have received responses from all the reviewers, and we greatly appreciate your involvement.** Your insights and suggestions have been instrumental in enhancing the quality and clarity of our work.