Perception of Knowledge Boundary for Large Language Models through Semi-open-ended Question Answering

Summary

This paper investigates the knowledge boundaries of Large Language Models (LLMs) using semi-open-ended questions, revealing limitations in LLMs' understanding. The authors introduce a novel method involving an auxiliary model to discover low-probability ambiguous answers, constructing a dataset of 953 questions across 32 domains. Their findings show that GPT-4 frequently produces unqualified answers and struggles with accurate self-evaluation, highlighting the need for improved detection of LLM knowledge boundaries.

Strengths

1. This work introduces a novel approach to evaluate LLMs using semi-open-ended questions, revealing limitations in existing methods focused on close-ended questions. The proposed method of identifying low-probability ambiguous answers is innovative and well-motivated. 2. Experiments show significant improvements in understanding LLM knowledge boundaries, particularly where models like GPT-4 may hallucinate, enhancing evaluation reliability. 3. The paper is well-written and easy to follow.

Weaknesses

1. The evaluation scope of this paper is limited, as it only assesses the performance of GPT-4 Turbo, which may be insufficient to represent the overall performance of LLMs. Have the authors analyzed other mainstream large language models such as Claude or LLaMA? It would be better to provide the performance of other large language models on this dataset. 2. The practical implications of the findings are not fully demonstrated. It is recommended to provide specific examples of applications to illustrate how these findings can improve the reliability of LLMs in practical use. Further explanation is needed on how improving training data or algorithms can mitigate these issues and enhance the practical utility of the models. 3. The paper mentions that GPT-4 performs poorly on semi-open-ended questions, but it may not have thoroughly analyzed the different types of errors (such as factual errors, logical errors, etc.) and their causes.

Questions

1. Has the author analyzed the performance of other large language models except GPT-4 on semi-open datasets? 2. Can you provide a more detailed analysis of the types of errors made by GPT-4 on semi-open-ended questions? 3. How do you ensure the reliability and consistency of human annotations in verifying ambiguous answers?

Rating

6

Confidence

3

Soundness

2

Presentation

3

Contribution

2

Limitations

Yes.

Summary

This paper focuses on detecting the “knowledge bounding” of the current large language models (LLMs), which would be helpful in handling the well-known hallucination problem in LLMs. In this paper, the authors explore in a new question answering setting (i.e. semi-open-ended questions). The authors employ LLM-based approach to construct semi-open-ended questions and obtain answers from a target LLM. Employing a open-source LLM, which is flexible to edit the parameters/variables, to detect black-box LLMs’ drawback. The proposed method that calculating the nearest semantic representation to select the related answers seems interesting. Finally, the paper not only constructs a dataset to find the knowledge boundary for GPT-4, but also indeed discovers 82.90% of GPT-4’s answers are not so satisfied. It also founds that 40.15% of its hard (ambiguous) answers generated are unqualified.

Strengths

1. The proposed method sounds novel for handling the knowledge boundary discover tasks. The method utilizes an open-source Large Language Model (LLM) to aid a black-box LLM, GPT-4, in identifying its knowledge limitations. They pinpoint words with low probabilities according to the model variable of the open-source LLM. Subtle modifications to the representation of the output layers facilitate the extraction of answers that are most analogous for LLMs. 2. The proposed semi-open-ended question task sounds important to explore. The task is more challenging than the current QA tasks, since the number of candidate answers and answer space (and may be the correct answers) is not fixed and deterministic. 3. The paper writing is quite clear and the paper organization is easy to follow. 4. This paper also fall into a popular and important direction. It tries to solve the effect of hallucination in LLMs in another way: different from case-by-case detecting the hallucination cases, this paper aims to discover the knowledge boundary of a given LLM (GPT-4). 5. The experiments are solid enough. It includes multiple base models (Llama with different size) and multiple evaluation metrics. The overall performance also includes study cases. 6. As GPT-4 is quite powerful, detecting the knowledge boundary of GPT-4 and find the shortcomings of GPT-4 is not an easy duty. The outcome of this paper is quite attractive, since it discovers 82.90% of GPT-4’s answers are not so satisfied with a simple LLama2 model.

Weaknesses

1. The proposed model works well on detecting the knowledge boundary in the setting of semi-open-ended question. Even if that setting makes sense, (1) I am not sure about how many questions in the real applications belong to “semi-open-ended question”. (2) how about the effectiveness of the proposed method on normal QA tasks (e.g. multiple-choice test). Could you please provide some insights on the proposed models’ potential strengthen on the normal QA tasks? 2. The model name used in the paper should by consistent. For example, the authors use both “LLaMA-2-13b” (in line 252) and “LLaMA-2-13B” (table 1) in the paper. 3. Typos: In line 252, it should be “we use two LLaMA-2-13b models”. The authors missed to add “model”. In table 1, the title of the second column should be “Auxiliary Model” instead of “Auxiliary Model Size”, since LLaMA-2-13B is a name instead of a kind of size. 4. What is the meaning of the underline results in Table 3? The authors should make it clear of each mark or notation in the paper is well-described.

Questions

1. Though the proposed model shows its effectiveness on semi-open-ended question setting, I am wondering the potential future work on normal QA tasks (e.g. multiple-choice test). Could you please provide some insights on the proposed models’ potential strengthen on the normal QA tasks?

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors' limitation section indeed addressed some concerns about this paper, which is fine with me.

Summary

In this submission, the authors aim to explore the detection of large language models’ (LLMs) knowledge boundary, which is a borderline to tell us what can LLMs really know. The detection of knowledge boundary wound play a crucial role to help the researchers to deal with hallucination. Different from the widely-used QA setting, the paper focuses on another QA setting (semi-open-ended questions), which may be ignored by other researches but seems common in our real life. To detect the knowledge boundary, the authors hire a small-scale open-source model (Llama) to help the GPT-4 to detect GPT-4’s boundary. This paper have some interesting findings. For example, GPT-4 underperform the auxiliary model, LLaMA-2-13B in 50% cases.

Strengths

1. The authors explore in a new question answering setting (i.e. semi-open-ended questions). This setting is unexplored in QA tasks around LLM, and it seems quite interesting and useful for LLMs. 2. Detecting the knowledge bounding is a promising and crucial direction to achieve a better usage of LLM and deal with the notorious hallucination problem. 3. The proposed idea makes sense and novel enough, which use a open-source LLM to assist the black-box LLM (GPT-4) to detect GPT-4’s knowledge boundary by accessing the prob dist and find the low-prob words. The minor changing of the representation of output hiddens helps to obtain the most similar answers for LLMs. 4. The paper writing is quite clear to follow with detailed graphical descriptions (the overview figure). 5. The performance of the proposed methods seems effective in detecting GPT-4’s knowledge boundary. It found that in some (50%) cases, GPT-4 underperform the auxiliary model, LLaMA-2-13B.

Weaknesses

1. The setting of “semi-open-ended questions” is quite interesting and seems useful and common in our real life. However, is the term of “semi-open-ended questions” widely used in the QA research area? Although I am not so familiar with with QA and its terminology, I suggest the authors conduct a fully survey of the related terms and explain more about the meaning of “semi-open-ended questions” in the next version. 2. The proposed method works well on the “semi-open-ended questions”. It would be better to give a detailed statistic (e.g. questionnaire) or empirical study or related analyses about “the proportion of semi-open-ended questions out of the whole QA scenario”. It shows the practical use of the proposed method in our real life. 3. Could you please illustrate the full version of the cases in Table 6 and Table 7 (Some cases are omitted due to the space limit)?

Questions

1. As mentioned in Weaknesses, could you please give a fully version of cases in Table 6 and 7?

Rating

8

Confidence

4

Soundness

4

Presentation

3

Contribution

4

Limitations

Yes, the limitation section addressed by the authors seems fine with me.

Summary

The paper presents a study of generating answers to questions from the tail of an LLM's distribution (GPT-4). They begin by constructing a dataset by generating questions with multiple answers from GPT-4. For each question, they continue decoding multiple answers and define the first 75% of generated answers as "common-sense answers" and the last 25% as "ambiguous answers". Their objective is to then to develop methods for generating such "ambiguous answers" from GPT-4 more efficiently, not relying on prompting many times (as they did to construct their dataset) or decoding parameters (which are not fully accessible for many API services). Their approach relies on using an auxiliary language model (LLama) to generate low-probability answers by reducing the probability of existing answers.They evaluate their method for generating low-probability answers using their constructed dataset, evaluating its ability to recover "ambiguous answers" as defined above. They conclude their work by performing analysis of these tail answers ("ambiguous" answers in their dataset and answers generated using their method), using retrieval+GPT-4 to verify each of these tail answers for correctness. They find that roughly half of these answers could were correct using thier system.

Strengths

This work develops a method for generating a large, diverse answer candidate set, which may be useful for a number our other tasks. The authors perform several ablation experiments, demonstrating the efficacy of different components of their method and sensitivity to hyperparameters.

Weaknesses

Methods for generating a diverse set of answer candidates are evaluated against their ability to recover the set of tail-answers from GPT-4. Analysis, however, demonstrated that roughly half of these tail-answers are incorrect. Evaluations are therefore designed to generate answers that match GPT-4's tail distribution and generating the same set of incorrect answers. The evaluation dataset is quite small and results only demonstrate minor improvement without significance testing. Many components of the work lack explanation. The authors reference "human annotation" for verifying answers; however, it is not clear how this is done, or what the exact instructions were. While the checklist notes that full annotation instructions and compensation details were provided in supplementary materials, but I did not find these. The retrieval-based system used for validating answers also lacks description, only noting that they used Microsoft Copilot to perform retrieval.

Questions

See the last point in weaknesses above. Also, its somewhat unclear to me how LLAMA model is used as an auxiliary model for GPT-4. Is it auxiliary because GPT-4 can be used to validate generated answers?

Rating

3

Confidence

4

Soundness

3

Presentation

2

Contribution

2

Limitations

See final note under weaknesses

Thanks for authors' responses

Thanks for your responses. I have read the authors' rebuttal, which has solved my concerns. After carefully considering the others' review comments and the author's rebuttal, I will keep my score.