Beyond Accuracy: Ensuring Correct Predictions With Correct Rationales

Summary

This paper introduces a novel method called dual-correct predictions, designed to train models to make accurate predictions based on correct rationales, thereby improving their safety for deployment. Additionally, the authors develop a unique dataset containing structured rationales that explicitly outline the reasoning processes required for visual recognition tasks.

Strengths

1. Authors constructed a new fine-grained rationale ontology dataset, which provides new solution ideas for generating faithful explanations 2. Authors propose a dual-correct method to improve the effectiveness of the models.

Weaknesses

1. Equation 4 lacks sufficient explanation. Especially the correct rationale. 2. Lack of algorithm flowchart to show the training and inference process of the model. 3. Authors do not have a more up-to-date explainable baselines. Other: Equation 4 and 5, missing in the last part ')'

Questions

1. What is the difference between Rationale ontologies proposed by the authors and prototypes in prototype learning? Is it considered to add prototype learning as baseline? 2. How does the authors' approach scale to a data without ontology labeling? 3. How does the model predict when the authors are reasoning? 4. Authors present the background of the problem in Figure 1, using GPT4V as an example. Why is it not in the experiment, comparing GPT4V?

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

See weaknesses and questions.

Summary

This paper proposes a method for incorporating rationales into the predictions of foundation models. The goal is to build models that make "dual-correct predictions": predictions that are correct because the model "reasons" using the correct rationale. To do this, the authors collect a new rationale dataset that captures additional information for each category in ImageNet. They then develop a method that faithfully attributes predictions to rationales for CLIP-VIT models, and propose an optimization approach that uses the rationale to guide the model during training. This method is then evaluated through a series of experiments. The authors find that their proposed method outperforms state-of-the-art models in both prediction accuracy and rationale correctness across image classification and retrieval tasks. As a result, the paper concludes that the proposed dataset, explanation method, and rationale-informed optimization approach provide a way to develop models with dual-correct predictions.

Strengths

I believe the strengths are as follows: - Problem goal: With the rise of foundation models being applied in increasingly impactful settings, it is imperative for them to not only make correct predictions but for them to do so for the right reasons; otherwise, they run the risk of failing in unpredictable ways. This paper does a nice job of discussing this problem and the goal of "dual-correct" prediction is intuitive and informative. I thought Figure 1 was a really nice figure. - Rationale dataset: The collected rationale dataset could be useful for future research that builds on rationale methods for ImageNet. - Experimental results: The paper conducts extensive experiments across a wide range of tasks, including image classification, retrieval, rationale localization, and disentanglement. The proposed method consistently outperforms other methods and ablated versions.

Weaknesses

Although the experimental results are strong, I think there are weaknesses that take away from the overall impact of the work: - Limited applications: The paper is framed in the title and intro as providing general methods for forming models with dual-correct predictions. However, the results are only for vision transformers: the faithful explanation method used is based on CLIP-ViT models, and the collected dataset is only relevant for object classification on ImageNet. Since the method and problem are framed generally, there should be more experiments across other domains (e.g. text) and models (e.g. diffusion models). - Using GPT-4 to generate gold-label rationales: The premise of the paper is that foundation models may not be using the right rationales when making predictions. However, their dataset of rationales is constructed using GPT-4, a foundation model. If we believe the premise (which I do), why should we trust GPT-4's generations as gold-label rationales? - Missing related work: There is a large literature in NLP that seek to develop methods that form sensible rationales (e.g. [1, 2, 3]). These should be included in the related work and there should be a discussion about how the method developed in the paper differs from them. - Clarity: Parts of the paper are confusing. For example I thought the rationale-informed optimization section was difficult to understand. There were also typos throughout the paper (e.g. "[33] averaging across layers and heads to get contribution of image token" in line 137 isn't a complete sentence). [1] T. Lei, R. Barzilay, and T. Jaakkola. 2016. Rationalizing neural predictions. [2] J. Bastings, W. Aziz, and I. Titov. 2019. Interpretable neural predictions with differentiable binary variables. [3] S. Gurrapu, A. Kulkarni, L. Huang, I. Lourentzou, L. Freeman, and F. A. Batarseh. 2023. Rationalization for Explainable NLP: A Survey.

Questions

See questions above relating to limited applications, trusting the rationales of GPT-4, and missing related work.

Rating

4

Confidence

3

Soundness

3

Presentation

2

Contribution

3

Limitations

Yes

Summary

Machine learning models are mostly commonly evaluated based on accuracy, yet highly accurate models can conceal issues with the underlying reasoning. For example, models can predict the right label while predicting it for the wrong reason. To improve the ability of foundation models to make good predictions for the right reasons, the authors contribute two items: 1) they develop a dataset where prediction labels are connected with a series of rationales, and 2) they develop a fine-tuning strategy to incorporate this into the model.

Strengths

1. **Formulation as rationales matches intuition** - The core idea of rationales is interesting as a building block for developing explanations. Such an idea seems to be present in different settings/scenarios, and matching up explanations with the underlying rationales seems to be an intuitive way to ensure that predictions are made for the right reason. 2. **The method is pluggable/usable across different models** - As pointed out in Section 3.3, the method proposed in the paper is usable across different architectures. Essentially, the proposed method is accomplished through optimization techniques rather than additional parameters or architecture changes. Because of this, the model proposed in the paper can easily be used with different models. 3. **Evaluation is across many datasets** - The impact of their fine-tuning method is evaluated against nine different datasets, which makes it clear what the results + impact is. In Figure 4, different images are shown comparing a CLIP baseline with the method proposed by the authors. Such a figure demonstrates the utility provided by their method, as they can improve heatmaps, which implies that they improve a model's ability to make predictions for the "right reason."

Weaknesses

1. **Unclear if the Dual-correct Prediction Problem is the same as the Right for the Right Reason** - Equations 4 and 5 sketch out the optimization problem that defines the dual-correct predictions, and how to optimize for right for the right reason. However, it's unclear if the optimization problem listed here is the correct one to pursue, and the reasons behind such a problem are unclear. It seems to be that far-apart rationales should still have dissimilar embeddings, and close-together rationales have similar embeddings, but such intuitions are not explained in the paper. More generally, the reason and structure of this optimization problem should be explained better, as it is unclear how or why such a problem connects with the overall issue of producing correct rationales. 2. **Generalizability of the dataset generation method is unclear** - The dataset generation method is focused on the ImageNet dataset, where different hierarchies are created based on the underlying ontologies. It is unclear whether such ontologies can be generated in non-ImageNet datasets, and how the location of such rationales can be determined in generality. Because of this, it's unclear whether the method to generate the structured rationale dataset can be widely replicated across domains.

Questions

1. Why is a tree structure used when designing rationales (line 46)? 2. Why does incorporating rationales also improve accuracy? 3. What is the difference between localizability and disentaglability?

Rating

6

Confidence

3

Soundness

3

Presentation

2

Contribution

2

Limitations

The authors acknowledge the limitations of their paper.

Summary

The goal of the paper is to train Vision Transformer models in a way that they make the "right predictions for the right reasons". The paper follows up on prior work in ML explainability that extracts "rationales" for model predictions. The idea of the paper (which is also considered in some prior works) is to align the rationale with the model predictions. The key issue that the paper faces is that rationales are quite difficult to label at a large scale. The paper gets around this problem by using GPT-4 based annotations where given a ImageNet class, 6 rationales in the form of a two level deep tree are generated. The paper then fine-tunes some Vision Transformer models to that maximize accuracy while disentangling rationales from each other. Experiments show an increase in accuracy and explainability performance.

Strengths

1. Explainablity of ML models is an important topic as it is critical for building trust in models. 2. The idea of disentangling rationales is interesting and looks like it should indeed lead to better explanations. 3. Experiments cover a wide range of datasets.

Weaknesses

1. The biggest issue with the paper is that it makes some choices without assessing their merit, which leads to a number of open questions. For instance, the paper uses GPT-4 to build the graph of rationales. How were the resulting rationales from GPT-4 evaluated? How were the examples that were passed to the model in Appendix A constructed? Did the paper study how well GPT-4 followed the instructions? For instance, some of the instructions are quite intricate. Consider for instance: “These features should be visually distinctable and have limited overlap with each other”. It is not clear how the model interprets “visually distinctable” and “limited overlap”. Did the model actually follow these instructions? 2. Continuing from the previous point, in line 180: How was the value of 6 chosen? What would be the effect of changing this value? Should all classes have the same number of rationales associated with them? Are some classes inherently "more complex" than others? Similarly, should all rationales be disentangled? One would assume that some rationales might be overlapping. For instance, a traffic light and a traffic light pole are overlapping whereas a traffic light and the road aren’t. 3. The paper also adds no discussion on the presence of rationales in the image. Are all the rationales (e.g., beak, tail) always visually present in every image of that class? Does it ever happen that the rationale is not visually present but the explainer still predicts it presence? Similarly, how does one ensure that the model “looks at” the rationales in the input when making the prediction? Could we actually blur that part and see if the model predictions / rationale still stay the same? 4. The writing of the paper is also quite rushed. Important details missing which make it hard to properly judge the merit of the paper. For instance, in line 158: What does the function $h$ actually look like? How does it connect to the vision and the language components mentioned in the previous paragraph? How is $c_i$ computed in Eq. 3 used later on? Similarly, in line 205: The rationales were extracted from GPT-4, so how is the localization ground truth (in terms of pixels in the image) obtained? In Figure 4: What exact procedure was used to get the heatmaps for CLIP explanations? How does the conversion of Eq. 4 to Eq. 5 via Lagrange multipliers solve the non-convexity issue? Isn’t the problem still non-convex? How exactly are the KKT conditions used here?

Questions

1. Please see the questions in points 1-4 in the "Weaknesses" section. 2. Minor nit: The example in Figure 1 seems a bit problematic. From a distance, the balloons could be taken as red lights in the shape of globes. Wouldn’t the correct question be: Is there a red *traffic* light in the image? 3. Eq. 3: Shouldn’t $w_l$ be inside the summation? 4. Table 2 caption: The last part of the caption is difficult to parse and looks like there is a typo here “our method also hence the prediction performance”. Is this phrasing intended? 5. Line 28: Looks like the base model predictions and the rationales (which often themselves are kinds of predictions) are two often independent things. Theoretically, is it possible to have models that always make the *right prediction* (100% generalization) but always with the *wrong rationale*? Could one train such models, e.g., via using some adversarial training? Would such models be trustworthy, even when one knows they are 100% accurate?

Rating

6

Confidence

3

Soundness

2

Presentation

1

Contribution

1

Limitations

The reviewer sees no societal issues. The paper does discuss some limitations, though there are many (see the Weaknesses section) that need discussion.

Followup Questions

Thank you for your rebuttal and comments. I had a couple of followup questions: 1. What is the relationship between equation 4 and finding the correct rationale? Moreover, why is D(h(...)) the correct mathematical formulation corresponding to having the correct rationale? 2. Do the results of the experiment imply that models not built upon spurious correlations would not be assisted by rationales (in terms of accuracy)? Moreover, is the role of incorporating rationales to alleviate the impact of spurious correlations?

Response to Reviewer VWuj

Dear Reviewer VWuj, Thank you once again for your valuable comments. To address your concern on the GPT-4 generated rationales, we further conducted **human evaluations**. In our previous rebuttal, we performed an automatic evaluation of the entire dataset, and the results indicated the high quality of our data. This human evaluation further proves that the machine evaluators are strongly aligned with human evaluators, confirming the reliability of our automatic evaluations and the high quality of our rationale dataset. Here are the details: \ **Evaluators:** \ **1) Human evaluators:** We recruited four human evaluators, who are mostly graduate students. They are asked to conduct assessments based on commonsense knowledge and perform Internet searches for validation. On average, it takes them one minute per sample. **2) Machine evaluators:** The latest GPT-4o and GPT-4v models. For each evaluation, we perform three independent runs and calculate the average scores. **Evaluation Metrics:** \ Factual Consistency: Whether the rationales are consistent with facts \ 5 - 100% consistent with fact \ 4 - 75% \ 3 - 50% \ 2 - 25% \ 1 - 0% consistent with fact (completely wrong) Comprehensiveness: Whether the rationales provide all information necessary to predict the category \ 5 - cover 100% of discriminative visual features \ 4 - cover 75% \ 3 - cover 50% \ 2 - cover 25% \ 1 - cover 0% of discriminative visual features Visual Disentanglement: Whether the rationales are visually disentanglable or non-overlap \ 5 - 100% of rationale visually non-overlap (completely disentangle) \ 4 - 75% non-overlap \ 3 - 50% non-overlap \ 2 - 25% non-overlap \ 1 - 0% of rationale visually non-overlap (completely overlap) **Evaluation Data:** \ We sample **three independent groups** of data from our rationale dataset, each consisting of 50 categories and their corresponding rationales. Specifically, categories were randomly selected from their superclasses: Animals (20), Objects & Artifacts (15), Natural Scenes (5), Plants (5), and Human Activities (5). This ensures not only each superclass is represented but also the robustness of our results [1]. [1] Torralba et al. "Unbiased look at dataset bias." CVPR 2011. **Quantitative Results:** | Evaluator | Factual Consistency | Comprehensiveness | Visual Disentanglement | |:---:|:---:|:---:|:---:| | GPT-4o | 4.89$\pm$0.05 | 4.55$\pm$0.06 | 4.66$\pm$0.06 | | GPT-4v | 4.92$\pm$0.03 | 4.67$\pm$0.05 | 4.70$\pm$0.02 | | **Machine Avg.** | **4.91** | **4.61** | **4.68** | | | | | | | Human_A | 4.85$\pm$0.11 | 4.64$\pm$0.19 | 4.42$\pm$0.15 | | Human_B | 4.97$\pm$0.02 | 4.77$\pm$0.02 | 4.20$\pm$0.11 | | Human_C | 4.78$\pm$0.04 | 4.60$\pm$0.11 | 4.78$\pm$0.10 | | Human_D | 4.81$\pm$0.08 | 4.64$\pm$0.05 | 4.77$\pm$0.07 | | **Human Avg.** | **4.85** | **4.66** | **4.54** | **Takeaway 1: The quality of our generated rationale dataset is high.** In the table above, we show the results on the average of three sample groups. The average scores demonstrate a high degree of agreement between machine and human evaluators. The dataset consistently achieves scores of 4.61 or higher on the average of machine and human evaluators for each metric, indicating that over 90.3% of the rationales for each category are highly factual, comprehensive, and visually disentanglable. **Takeaway 2: Our automatic evaluation is reliable and scalable.** As shown in the table above, the average scores of each metric are almost identical between machines and humans. The Pearson correlation coefficient of 0.82 reveals the strong positive correlation between machine and human evaluations. Therefore, using the automatic evaluation method can efficiently evaluate the entire dataset, and align with human evaluations. Note that the results of the entire dataset are reported in the previous rebuttal. **Qualitative Results:** \ Here we list some categories that rated high in automatic evaluation. As shown, all the rationales are consistent with fact, sufficient for distinguishing the corresponding category, and visually non-overlap (disentangled). \ "ostrich": ["Long, bare neck", "Large, powerful legs", "Small, feathered head", "Two-toed feet", "Black-and-white feathers", and "Long, curved beak"] \ "tabby cat": ["Striped fur pattern", "Dark "M" on forehead", "White paws/chest", "Dark ears/tail tip", "Round face shape", "Whiskers/eyebrows"] \ "airliner": ["Long, slender body", "Multiple engines", "High wingspan", "Narrow fuselage", "Tail fin", "Multiple windows"] We sincerely appreciate the time and effort you have invested in providing us with your constructive feedback. We eagerly look forward to hearing from you and are fully prepared to address any additional questions or concerns you may have.

Response to Reviewer dtoB

Dear Reviewer dtoB, We sincerely appreciate the time and effort you have invested in providing us with your encouraging feedback. Your insightful comments are invaluable to the enhancement of our work. We eagerly look forward to your feedback and are ready to address any additional questions or concerns that might arise. Thank you once again for your valuable contribution. Best wishes, Authors