Refine Knowledge of Large Language Models via Adaptive Contrastive Learning

Response to reviewer ebFf

We greatly appreciate the valuable feedback from the reviewer, and hope our response can address your concerns. > Weakness: In this work, the discussion surrounding the concept of "I know that I don't know" (Quadrant 2) is lacking. We apologize for the confusion with our description of “I know that I don't know”. In our work, we believe that when LLM gives a correct answer to a question as a response, it is regarded as “I know that I know”. When sampling responses, a question is considered as “I know that I don't know” when the model not only responds “I don't know”, but also satisfies the condition that it is really beyond the model's knowledge. In fact, it is possible for a model to respond “I don't know” to a question for which it knows the correct answer. At this point, we consider the model “I don't know that I know”. Whether or not the model actually understands the answer to the question is determined by the response accuracy of our sampling. It is worth noting that, as we sample responses, we observe that without additional prompts or fine-tuning, the LLM tends to give a response even if it doesn't know the correct answer, so it almost never chooses to reject a response. In our experiments, we added an additional prompt that told LLM to reply “I don't know” to questions it did not know. At this point, we regard those questions with the response “I don't know” that are labeled as questions for which the model really did not know the corresponding knowledge in our data construction as the model's “I know that I don't know”. > Question: How does varying the inference temperature affect the model confidence in your setting? Thank you very much for your suggestion. We refer to past work [1][2] and note that they do not explore different temperatures, but rather use constant tempareture for reasoning. **Therefore, for the sake of a fair comparison, we choose the same temperature value of 0.7 as theirs, without additional modifications and further exploration.** [1] [R-Tuning: Instructing Large Language Models to Say ‘I Don’t Know’](https://aclanthology.org/2024.naacl-long.394) (Zhang et al., NAACL 2024) [2] [Can AI Assistants Know What They Don't Know?](https://proceedings.mlr.press/v235/cheng24i.html) (Cheng et al., ICML 2024)

Response to reviewer 5ff6

We greatly appreciate the valuable feedback from the reviewer. To address the reviewer's concerns, we have conducted additional experiments: > Weakness 1: First, the experiment coverage is relatively narrow, with results demonstrated on only two datasets. Although the method shows promising results, its broad applicability across more tasks and domains is uncertain. Expanding the evaluation to more diverse datasets, such as dialogue or fact-checking datasets, would help validate the robustness and applicability of the method. **Response** In addition to the two experiments mentioned in the original text, **we have conducted an additional experiment using the LLaMA-2-7B-chat model on the Alcuna dataset.** This dataset is designed by creating artificial entities through the alteration of existing entity attributes, leading to the generation of questions about these novel entities. Since these entities are artificially constructed, it is nearly impossible for the model to possess prior knowledge about them, making this dataset ideal for testing the model’s ability to refuse to answer inconceivable queries. We conduct this experiement to evaluate the model's ability to respond appropriately to completely unknown questions. | | ALCUNA | | | |-------------------|-----------------------|-------|----------| | | IK-IK | IK-IDK| TRUTHFUL | | IDK-Prompting | 1.6 | 90.3 | 91.9 | | IDK-SFT | 1.2 | 96.6 | 97.8 | | IDK-SFT-Adpt-Ctr | 1.0 | 97.3 | 98.3 | As we can see in the table, our proposed methods perform better in the context of unknown queries comparing with the models that finetuned with IDK-SFT. The results show that the proposed method effectively avoid giving misleading answers, proving reliability when dealing with unknown areas. **These findings have be updated in Appendix C.1 of our revised paper.** > Weakness 2: Second, although the paper proposes using a combination of three loss functions for different types of knowledge, it does not delve into the interactions and balance between these loss functions. In different scenarios, each loss may contribute differently; exploring the interactions and balance of these loss functions across various tasks would further strengthen the theoretical foundation of this method. **Response** We sincerely appreciate the reviewer's insightful comments regarding the interaction and balance between the loss functions. **In response to your feedback, we have designed experiments to explore these interactions. Specifically, we utilized our Adaptive Contrastive Learning strategy to test different combinations of our loss functions.** We paired the three loss functions in two's to observe their interactions. The results of these experiments are presented in the following table: | Loss Combination | TriviaQA | | | |----------------------------------------|---------------|-------|----------| | | IK-IK | IK-IDK| TRUTHFUL | | $L_{Q_1}$ | 28.7 | 46.0 | 74.7 | | $L_{Q_3}$ | 30.0 | 43.8 | 73.8 | | $L_{Q_4}$ | 36.5 | 39.3 | 75.8 | | $L_{Q_1}$ + $L_{Q_3}$ | 26.5 | 47.9 | 74.4 | | $L_{Q_1}$ + $L_{Q_4}$ | 32.9 | 45.1 | 78.0 | | $L_{Q_3}$ + $L_{Q_4}$ | 29.4 | 46.7 | 76.1 | | Total | 37.3| 40.9 | 78.2 | In this table, $L_{Q_1}$ indicates "Model knows it knows'' in $Quadrant-1$, $L_{Q_3}$ indicates "Model doesn't know it doesn't know'' in $Quadrant-3$, and $L_{Q_4}$ indicates "Model doesn't know it knows'' in $Quadrant-4$. For $L_{Q_1} + L_{Q_3}$, the model has a higher IK-IDK rate due to cautious responses, but lower IK-IK compared to Total. Missing $L_{Q_4}$ limits the use of underlying knowledge for confident answers. With $L_{Q_1} + L_{Q_4}$, the model sees strong Truthful rate and decent IK-IK results. However, without $L_{Q_3}$, it's less cautious about unknowns, affecting the IK-IDK score. The $L_{Q_3} + L_{Q_4}$ combination provides balanced performance but lacks the strong IK-IK scores of Total. This suggests that without $L_{Q_1}$, the model struggles with certainty in known answers. **These findings have be updated in Section 5.2 of our revised paper.** Reference: [1][ALCUNA: Large Language Models Meet New Knowledge](https://aclanthology.org/2023.emnlp-main.87) (Yin et al., EMNLP 2023)

Response to reviewer BPiT (part 1)

Thank you very much for your recognition of our work and your insightful comments. We hope to address your concerns through our response: > Weakness 1: Even though the results look strong by looking at the Truthful Rate, I am not sure whether the proposed Truthful Rate is well-defined. Given the denominators of IK-IK Rate and IK-IDK Rate are different, I don't think it's rigorous to add them together as the final score. An alternative could be treating answering "I don't know" to those answerable questions as correct and computing the overall accuracy on the full dataset. Also, why the IK-IK rate for Mistral-7B-Instruct-v0.1 drops drastically compared to IDK-prompting? This is a bit concerning as it indicates after Adaptive Contrastive Learning, the model may not be useful anymore. **Response** - **For the definition of evaluation metrics:** we apologize for confusing you due to unclear definition of evaluation metrics. In fact, the IK-IK Rate (I know what I know rate) represents the proportion of questions answered correctly by the model out of all questions. The IK-IDK Rate (I know what I don’t know rate) represents the proportion of questions that the model correctly refuses to answer, out of all questions. **Therefore, the denominators of the IK-IK Rate and IK-IDK Rate are the same. We have clarified this in Section 4.3 of our revised paper**. Furthermore, the Truthful Rate is the sum of the IK-IK Rate and IK-IDK Rate. It represents the proportion of questions for which the model provides truthful responses. *The higher the value of Truthful rate, the clearer the model’s perception of what it knows and does not know, which also indicates a higher level of truthfulness.* - **For the IK-IK Rate for Mistral-7B-Instruct-v0.1:** Considering our used three evaluation metrics, the higher these three metrics are, the better. Among these metrics, we argue that the Truthful Rate is the most important one because it indicates the probability that users will receive a truthful response. Therefore: - From the perspective of overall performance, **Adaptive Contrastive Learning has obvious advantages over IDK-prompting and IDK-SFT, which indicates that Adaptive Contrastive Learning will allow the model to produce more truthful responses**, which is relatively more user-friendly, because users certainly don’t like the model not knowing what they know (i.e., catastrophic knowledge forgetting) or not knowing what they don’t know (i.e., knowledge hallucination). - Looking at the IK-IK Rate and the IK-IDK Rate separately, **we think that we should not only see the decrease in the IK-IK Rate, but also the increase in the IK-IDK Rate, and the increase in the IK-IDK Rate is greater than the decrease in the IK-IK Rate**. This shows that after the Adaptive Contrastive Learning's optimization, the model has become more cautious because it knows more about what it does not know, and even answers the knowledge it knows more cautiously. Of course, as mentioned in the above point, we think this is friendly to users. After all, at least the model does not answer the hallucination knowledge that it does not know. > Weakness 2: I am unsure what's the major takeaway of Section 5.3. Does the accuracy under repeated samplings represent the concept of "confidence"? Actually, the paper mentions the term of "confidence" when introducing the motivation of the proposed method. It would be interesting to see whether the proposed method can draw a more rigorous connection of model's confidence in its generation. **Response** We sincerely apologize for the confusion caused to you by Section 5.3 and Figure 4. **First of all, we need to clarify that the vertical axis in Figure 4 represents the proportion of samples that fall into a specific accuracy range in the entire data set.** Specifically, in Section 5.3, we used repeated sampling to make both models answer each question in the TriviaQA test set 10 times. We used the number of correct answers out of the 10 responses (i.e., accuracy) to reflect the performance (or confidence) of the model. Therefore, to explain further, for example, in the “Unknown Questions” sub-figure of Figure 4, the bars with the accuracy of 1, their specific values indicate the proportion of questions that the models correctly rejected to answer 10 times, out of the entire dataset. For example again, in the “Known Questions” sub-figure of Figure 4, the bars with the accuracy of 0.8, their specific value indicate the proportion of questions that the model answered correctly 8 out of 10 times, out of all questions. **We have rewritten Section 5.3 to eliminate potential confusion for readers.**

Response to reviewer BPiT (part 2)

> Weakness 3: The presentation for the paper can be further improved. See my detailed comments below. **Response** Thank you very much for your valuable suggestions, which can make the readability of our paper better, and we have revised them one by one in the updated version according to your suggestions. **Please note that you can see the updated version we uploaded on the OpenReview website.** *To make it more convenient for you to review our revised version, we have highlighted our modified content in blue.* > Question 1: In Table 1, it's better to use "IDK" instead of "idk" to be accord with the main paper. **Response**: Yes, we have changed to use 'IDK' in Table 1. > Question 2: I suggest rewriting Section 5.2 as the current writing is a bit confusing, even though I can get the meaning by reading Table 3. For example, for the first bullet point, "With" should be "Without" based on my understanding. **Response**: We sincerely apologize for the confusion caused to you by Section 5.2, especially Table 3. We have optimized the writing of Section 5.2 and Table 3 in our revised paper. > Question 3: I am wondering if the notations of the paper can be improved. For example, using $IDKIK$ as subscript in equations looks weird. **Response**: Thank you again for your suggestions. We have simplified the notations of our paper. For example, we use ${Q_1}$, ${Q_2}$, ${Q_3}$ to represent Quadrant-1, Quadrant-2, Quadrant-3 in the revised paper.

Response to reviewer tNNV

Thank you for your valuable feedback and for pointing out areas for improvement. > Weakness: The experiments in the paper are not solid enough, only on LLAMA2-7B, and more different models such as Qwen2.5 series should be considered to enhance the said reliability of the experimental results. And the effect on different size models should also be considered and analyzed, such as 13B and above. The baseline in the paper is too simple and lacks comparison with other up-to-date approaches to knowledge boundary identification. **Response** - While our primary model was indeed LLaMA-2-7B, it was not the only model we used. We also conducted experiments with the Mistral-7B-Instruct-v0.1 and obtained promising results. - We agree that evaluating different model sizes is crucial. **Given our current resources and timeline, we performed additional experiments on LLaMA-2-13B-Chat**. The results are summarized in the following table: | | LLaMA-2-13B-Chat | | | |-------------------|-----------------------|-------|----------| | | TriviaQA | | | | | IK-IK | IK-IDK| TRUTHFUL | | IDK-Prompting | 37.7 | 31.6 | 69.3 | | IDK-SFT | 32.8 | 41.3 | 74.1 | | IDK-SFT-Adpt-Ctr | 37.8 | 41.1 | 78.9 | From the table, we can observe that our method remains competitive even with larger model sizes. **These findings have be updated in Appendix C.2 of our revised paper**. - We acknowledge the importance of testing other models like Qwen2.5. Unfortunately, due to its recent release, we could not include it in the current version of our paper on time. However, we fully agree with your suggestion and plan to incorporate Qwen2.5 in future experiments to validate our approach across different models. > Question: Since the use of the proposed method enables the model to distinguish knowledge boundaries effectively, it would be interesting to study the performance of the model after using the RAG technique. For example, comparison experiments with paper Enhancing Noise Robustness of Retrieval-Augmented Language Models with Adaptive Adversarial Training (Fang et al., ACL 2024) could be added. **Response** Regarding the suggestion of incorporating the RAG technique, we have used a benchmark RAG-Bench publicly available from the paper mentioned above[1]. RAG-Bench provides specific contexts for each query for the model to reference and respond to. In this experimental setup, we input the context alongside the query into the model (denoted as ``with RAG" in the table) and compared the results with those obtained without using context. | | RAG-Bench | | | |-------------------|-----------------------|-------|----------| | | IK-IK | IK-IDK| TRUTHFUL | | IDK-Prompting | 59.7 | 0.1 | 59.8 | | IDK-Prompting with RAG | 63.0 | 5.40 | 68.4 | | IDK- SFT | 60.3 | 1.90 | 62.2 | | IDK-SFT with RAG | 66.1 | 3.30 | 69.4 | | IDK-SFT-Adpt-Ctr | 44.5 | 22.4 | 66.9 | | IDK-SFT-Adpt-Ctr with RAG | 58.4 | 12.8 | 71.2 | As evident from the results, all methods benefit from the addition of RAG contexts, and our fine-tuned models exhibit a notable performance boost of 1.8 compared to SFT when utilizing RAG. This demonstrates that RAG enhances model outcomes and that our fine-tuning method integrates effectively with RAG. **We have updated these RAG-related experimental results in Appendix C.3 of our revised paper.** Reference: [1] [Enhancing Noise Robustness of Retrieval-Augmented Language Models with Adaptive Adversarial Training](https://aclanthology.org/2024.acl-long.540) (Fang et al., ACL 2024)

Friendly reminder to reviewer tNNV

Thank you very much for your insightful suggestions and comments. **Based on your advice, we have analyzed the experimental phenomena under the RAG scenario and have updated our revised paper accordingly**. We hope these analyses can address your concern. Considering that the rebuttal discussion phase of ICLR is about to end, we sincerely hope to receive your further feedback so that we have the opportunity to continue our discussion with you. Once again, thank you for your hard work and selfless help.