Forcing Diffuse Distributions out of Language Models

Summary

This work finds that (1) instruction-tuned language models fail to replicate high-entropy distributions over, e.g., random numbers (roll a 6-sided die) or names (give me a randon name). Further, it finds that (2) finetuning on a sample-approximation of the target distribution allows for higher-entropy distributions closer to the target distributions, and (3) that finetuning on some such requests even makes the instruction-tuned model output higher-entropy distributions for other requests (transfer, e.g, “give me a name” to “give me an occupation) while (4) not degrading the language model’s capabilities too much in general as measured by MT-Bench. The last point here was quite surprising to me, and only mentioned in passing in the main text, but is very important! Instruction-tuning leads to very low-entropy distributions, and so these entropy-encouraging finetuning I might’ve expected to break various things. I think this should be explored more in the final draft. The paper conducts experiments in generating baby names, countries, fruits, days and dates, numbers, occupations, and synthetic biography generation. They do some out-of-distribution extrapolation experiments that are particularly convicing, e.g., from one task to another, or from some range of numbers (finetune on 0-10) to another (test on 100-140).

Rating

7

Confidence

4

Ethics flag

1

Reasons to accept

The problem of this paper is well-motivated, and the method is nice and simple (though, see reasons to reject about the framing.) I was a bit cautious at first when reading the experiments, but the various out-of-distribution experiments convinced me that there’s a surprising and useful amount of generalization from the simple finetuning done and the range of generation prompts for which one might get better distributions. It’s a clearly written and concise paper. Overall, I just think this is a nice contribution that’s well-explored using various OOD generalization experiments; a simple method that seems to work and will be of interest to a broad range of people. I have some problems with the framing and experimental design below, but I think it should be accepted and people will learn interesting things from it.

Reasons to reject

The framing of this paper is that it’s a new method for encouraging diffuse distributions, but the method more or less is to finetune on samples from the desired distribution – e.g., if you want to match a 6-sided die, sample from a 6-sided die and train the model to output the desired answer after the proper prompt. For sampling baby names, sample from (some type of) distribution over names and then finetune. This is what all finetuning does when trying to match any distribution. So, “we propose a finetuning method” is a bit overmuch to me. I’d strongly encourage the authors to re-frame to emphasize that the simple–I’d even say default–method of just finetuning works way better than one might expect. There are considerable details missing from the paper. For example, the learning rate, number of epochs, etc. of the finetuning, for each model. This is really important since, e.g., too much finetuning would certainly eventually break the generalization of the model. The framing of the paper also makes blanket claims about language models, but only considers instruction-tuned models. I’d just change the language – wherever you say “language models” fail at X, please consider saying “instruction-tuned language models” fail at X. Instruction-tuning destroys model entropy, and the claims you make are not verified to hold for pretrained models. I think the “models don’t degrade too much elsewhere” really should be explored more; this is crucial for it to be a useful method! At least explain more of the MT-bench results in the main paper.

Summary

The paper investigates how to increase diversity in LLM generations. The method is simple. It assumes some probability distribution over a fixed set of outcomes (e.g., 6-sided dice, uniform). It then minimizes weighted cross-entropy loss to match model probabilities with the ground-truth. Given two toy tasks and a more realistic "biography generation" task, experiments show unnatural biases in instruction-tuned LLMs (Gemma, Llama-2, Mistral). The proposed method effectively mitigates those biases. Two interesting and novel results: (i) this fine-tuning does not seem to degrade other abilities of the LLMs, (ii) there is cross-task generalization where increasing diversity in one attribute also increases diversity in others.

Rating

7

Confidence

4

Ethics flag

1

Reasons to accept

- [S1] The method is simple to apply and appears effective in toy tasks and a more realistic "tabular" data generation task. - [S2] Results are validated on a range of LLMs, all show similar trends. - [S3] The paper is well written and the results are well presented.

Reasons to reject

I do not have any major concerns about this paper and recommend acceptance. The below concerns are minor. - [W1] The tasks considered are relatively simple. Synthetic data generation with a more open-ended output space are probably not feasible with this proposed method. However, this limitation is acknowledged in the conclusion. - [W2] Missing detail on fine-tuning setup (Sec 3.2). It would be important to mention for how many steps fine-tuning with LoRa was applied. What was the stopping criterion? - [W3] Missing detail on tasks (Sec 5): I could not find what were the precise empirical distributions for each of the tasks. Could be added to appendix.

Questions to authors

Typos: - pg 6. transferrability --> transferability - pg 7. "on all tasks in Section 3" --> Section 5? - pg 8. "gender are birth" --> "gender and birth"

Summary

This paper addresses the problem that LLMs perform poorly at generating diverse outputs, even when explicitly instructed to do so. The authors propose a fine-tuning method that encourages language models to output more diffuse distributions over valid choices. They use a parameter-efficient LoRA approach for the fine-tuning. Through experiments on toy tasks like generating random numbers, baby names, countries, etc., they show that the fine-tuned models generate significantly more diverse outputs that better match the expected distributions compared to the baseline models. Also, it seems the models exhibit strong generalization, i.e., a model fine-tuned on a subset of tasks generates diverse outputs even for held-out tasks. Applying the method to a more complex synthetic biography generation task leads to outputs with substantially more diverse attributes like names, birthplaces, occupations, etc. The general capabilities of the models (tested on a benchmark) are preserved after the fine-tuning for diverse outputs.

Rating

7

Confidence

4

Ethics flag

1

Reasons to accept

- The authors quantitatively analyze the mismatch between language model outputs and user expectations in terms of randomness and diversity. - They introduce a fine-tuning method that improves the diversity of language model outputs across valid options while maintaining generation quality. The proposed method is flexible and can handle tasks where the set of valid options is not easily enumerable. The fine-tuned models demonstrate strong generalization capabilities, meaning models trained on one set of tasks can generate diverse outputs for different, unseen tasks. The generalization property allows the method to enhance diversity in complex generation settings, such as synthetic dataset creation. - When applied to generating synthetic biographies, the proposed approach significantly increases the uniqueness of generated attributes. The improvements in diversity are achieved without the need for complex prompt engineering, decoding strategy adjustments, or manual editing, simplifying the process of generating diverse outputs.

Reasons to reject

- The authors' proposed approach is fundamentally aligning the output token probabilities with the desired ground-truth (random) probabilities. However, direct operations on token probabilities can also be applicable, like adding bias on the candidates (~200 selected by the users). This can be combined with higher temperatures to flatten the distribution of candidates and avoid the situation of degeneration. - The experimental results are mostly introducing the phenomenon without further discussion and analysis. Like the claimed strong cross-task generalization ability in Section 5.1. I would be more likely to know about the potential reason for this interesting phenomenon. - The motivation and the methodology are somehow independent of each other. To solve the issues mentioned in the motivation, there should be a bunch of prompting methods: Directly prompting LLMs to generate 500 numbers between 0 and 1, or attributed prompting for diverse dataset creation, etc.

Questions to authors

- I am really curious about the potential application or further impact of this work. - As the authors have some theoretical support for their fine-tuning method to minimize the distribution between the model outputs and ground-truth, is it possible for the authors to offer some proof that the tuning method will not influence the generation abilities of LLM (phenomenon in Appendix B.5).

Summary

This paper adresses an important issue of LLMs: their lack of diversity in generation. In the worst case, LLM fail to generate random answers when it should be possible. As a solution, the authors proposes a fine-tuning objective as workaround: the goal of this criterion is to have less peaky distribution to generate more diversity. The implementation relies on a low rank (LoRA) approach. Experiments mostly use toy tasks for illustration purpose.

Rating

4

Confidence

4

Ethics flag

1

Reasons to accept

- The addressed issue is well motivated. - The capacity of LLM to generate diverse output is an important goal

Reasons to reject

As I said, the problem is important. However, it is addressed as a fresh new problem while it is not. The related work section is too short and too confused. For instance MAUVE paper published in NEURIPS 2021 has initiated an interesting discussion on the evaluation of LLM generation, including the lack of diversity. The discussion is still lively nowadays. See for instance https://www.jair.org/index.php/jair/article/view/13715 The second concern is the description of the method. Equation 5 just describes a cross entropy term and the surrounding text does not clarify the novelty. I think I understand that the criterion align the distribution of the predictions with the expected one. In this case, it should be better explained, because the cross entropy is not new. At the end the experimental part does not provide meaningful results that could clarify the contributions.

Thanks

Thanks the authors for the rebuttal. Most of my concerns have been resolved. I have increased my score accordingly.

Author-Reviewer Discussions

Dear reviewers, Thank you for your review of this submission. The authors have provided their rebuttal, and it is time to engage in discussions with them if you have additional questions. Please take the time to read the other reviews and update your own review as necessary. Reviewers Jrar/Zz3Z: Even if you do not have further questions and/or choose to maintain your original score, please acknowledge that you have read the authors' rebuttal. Thank you! AC