STaR-GATE: Teaching Language Models to Ask Clarifying Questions

Summary

The paper tackles the challenge of language models' ability to ask effective questions to understand user preferences, crucial for personalized interactions. It introduces STaR-GATE, a method combining active preference elicitation with self-improvement, aiming to enhance the model's questioning strategy. The paper introduces of a synthetic dataset of 25,500 unique persona-task prompts, and demonstrates its effectiveness in improving the model's ability to generate personalized responses. The paper's findings underscore the significance of teaching language models to ask better questions, potentially impacting various domains like healthcare and education by improving dialogue effectiveness.

Rating

6

Confidence

4

Ethics flag

1

Reasons to accept

* **S.1.** The paper is well written, contains a well detailed background and related work, and the illustrations are helpful and informative. * **S.2.** The proposed STaR-GATE algorithm outperforms the baselines on the newly proposed dataset. * **S.3.** The STaR-GATE is shown to generalized to new tasks and new user preferences.

Reasons to reject

* **W.1.** The paper lacks evaluation comparison to existing works such as OPEN [1]. * **W.2.** The evaluations are conducted on single fine-tuned model and a single synthetic dataset. * **W.3.** The paper lacks novelty and seems to be a combination of existing works [1][2] with similar filtering approach to that of ToolFormer [4]. [1] Handa, K., Gal, Y., Pavlick, E., Goodman, N., Andreas, J., Tamkin, A. and Li, B.Z., 2024. Bayesian preference elicitation with language models. arXiv preprint arXiv:2403.05534. [2] Zelikman, E., Wu, Y., Mu, J. and Goodman, N., 2022. Star: Bootstrapping reasoning with reasoning. Advances in Neural Information Processing Systems, 35, pp.15476-15488. [3] Li, B.Z., Tamkin, A., Goodman, N. and Andreas, J., 2023. Eliciting human preferences with language models. arXiv preprint arXiv:2310.11589. [4] Schick, T., Dwivedi-Yu, J., Dessì, R., Raileanu, R., Lomeli, M., Zettlemoyer, L., Cancedda, N. and Scialom, T., 2023. Toolformer: language models can teach themselves to use tools. 2023. arXiv preprint arXiv:2302.04761.

Questions to authors

* **Q.1.** How would STaR-GATE handle scenarios where follow-up questions are not required at all. * **Q.2.** Why was the limit of the conversation length (the number of turns) chosen to be $K=3$? * **Q.3.** Does it make sense to force the conversation length to be uniform? In practice, do longer conversation, that is more follow up questions, lead to higher win rate?

Summary

The paper focuses on improving the ability of language models to ask clarifying questions to enhance response quality when faced with ambiguous user requests. The study introduces STaR-GATE, a method that involves iterative finetuning of a language model called the Questioner, which interacts with a Roleplayer to elicit user preferences. The approach is based on generating and finetuning on useful questions that lead to high-quality responses.

Rating

9

Confidence

4

Ethics flag

1

Reasons to accept

1. Good novelty: the paper presents a novel integration of active preference elicitation with a self-improvement feedback loop to refine the model's questioning capabilities. 2. Good results: The refined Questioner generates responses preferred over the initial model's responses in 72% of tasks after two iterations. 3. Contribution in datasets and open source code: the study involves a synthetic dataset of 25,500 unique persona-task prompts, and the authors have the intention to open source their findings. 4. Significant impact for interactive applications using language models: this approach can significantly enhance the performance of language models in user interaction scenarios where the user's preferences are not explicitly stated, such as in personalized services or customer support systems. 5. Fair set of ablation studies: the paper includes several ablation studies to explore different aspects of the training methodology and the effects of various design choices on the performance of the language model.

Reasons to reject

1. The data used is small and synthetic: it may not fully capture the complexity of real-world interactions. 2. The oracle is also an LM: GPT-4 is great but it may not be as good as real-world interactions. 3. Lack of discussion on the negative impact of fine-tuning: what did the 7B models lose to become good in asking interactive questions? Note: it's worth noting that most of these weaknesses are either scability issues or difficult problems which are worth studying as independent papers, which are understandably beyond the feasibility of this single paper.

Questions to authors

In a fune-tuning process, a language model (especially when it's small, like 7B) becomes good in the prescribed tasks but may lose performance in other scenarios. Could you discuss whether this phenomenon applies to STaR-GATE? What did the model lose to become good at asking interactive questions? What benchmarks can be used to measure this?

Summary

The paper presents "STaR-GATE," a novel framework designed to enhance language models' (LMs) ability to ask clarifying questions, thereby improving their response quality by eliciting more relevant information from users. The methodology employs a synthetic dataset and iterative self-improvement techniques, integrating active preference elicitation with response regularization to refine question asking. The results demonstrate significant improvement over baseline models in generating responses that are preferred by users, indicating the model's increased effectiveness in understanding and addressing user preferences through targeted questioning.

Rating

6

Confidence

3

Ethics flag

1

Reasons to accept

Innovative Approach: The STaR-GATE framework's combination of active preference elicitation and self-improvement represents a significant advancement in teaching LMs to ask more effective questions. This approach addresses a critical gap in current LM capabilities, particularly in dealing with task ambiguity. Robust Methodology and Empirical Validation: The paper provides a thorough experimental setup with a well-constructed synthetic dataset of persona-task prompts. The iterative training process and the inclusion of regularization to avoid overfitting on elicitation are both well-justified. The empirical results showing a 72% preference for responses from the STaR-GATE fine-tuned model over the baseline are compelling. Generalization and Practical Relevance: The ability of STaR-GATE to generalize across different role players and its applicability in practical scenarios, such as personalized user interactions, are particularly noteworthy. This enhances the framework's potential for real-world applications.

Reasons to reject

Complexity and Computational Efficiency: The paper does not fully address the computational demands of the STaR-GATE methodology, particularly regarding the scalability of its iterative fine-tuning process. For practical deployment, understanding the computational overhead and efficiency is crucial. Dependence on Synthetic Data: The reliance on a synthetic dataset might limit the model's effectiveness in real-world scenarios where user inputs are more varied and less predictable. The paper could benefit from additional experiments involving real user data to validate the approach's efficacy in naturalistic settings.

Summary

This paper describes an approach to generation of clarification questions intended to resolve issues of task ambiguity in task-oriented dialog. Training is performed by self-play on generated data comprising 25K persona prompts and takes the form of simulated role playing to elicit preferences. Two iterations of such self-training yields substantial improvements in response preference as measured by success in simulated tasks. The paper claims that the finetuned model generalizes to new personas. It is a little hard to evaluate the claims in this paper, because it does not step out of the synthetic mode of training and evaluation. It is therefore not possible to correlate the evaluation with some form of user experience in a real-world task. It may nevertheless prove useful in an application scenario. [I have read the authors' response.]

Rating

7

Confidence

4

Ethics flag

1

Reasons to accept

This is a well-implemented description of an approach designed to improve the capacity of a model to ask clarification questions in task-oriented dialog. The training and evaluation processes are documented in detail and should be easy to reproduce with some degree of success.

Reasons to reject

Empiricism, Data, and Evaluation: It is a pity there is no attempt at human evaluation, because this paper has an HCI dimension that is not addressed. Evaluations are entirely performed on synthetic data, and are rigorously defined. But it is hard to evaluate the impact of this form of model training in application scenarios. For example, it is hard to know if self-play training is actually successful at generating clarification questions that to a human user would seem well-timed and relevant. There is no way of knowing whether a clarification question is actually filling a gap in the system’s knowledge. A constant barrage of inappropriate clarification questions may, for example, irritate a user to the point of being counter-productive. It is still unclear to this reviewer at the end of the day what triggers the clarification question. How is it determined that a question needs to be asked at that particular juncture in the conversation. Some well-motivated work on neural question generation is neglected: • Rao and Daume 2018. Learning to Ask Good Questions: Ranking Clarification Questions using Neural Expected Value of Perfect Information. • Rao and Daume. 2019. Answer-based adversarial training for generating clarification questions • Aliannejadi, 2021 Building and Evaluating Open-Domain Dialogue Corpora with Clarifying Questions. EMNLP An early work in the context of questioning of images that is somewhat relevant Is Mostafazadeh et al 2017 Image-grounded conversations: Multimodal context for natural question and response generation. This work uses follow-on questions in Twitter conversations to attempt to learn questions about content that was not explicitly shown in the image.