Smoothie: Label Free Language Model Routing

Summary

This paper proposes a label-free routing method, Smoothie, to route an ensemble of LLMs without annotated data. Smoothie constructs a latent variable graphical model over semantic embedding representations of observable LLM outputs and the unknown ground truth, estimates the sample-independent quality scores of each LLM, and routes to the LLM with the highest quality score. Experimental results indicate that Smoothie competently performs several generation tasks over the 3B and 7B ensembles.

Strengths

1. Smoothie utilizes a latent graphical model and an embedding language model to estimate the quality of each separate LLM without labels, thus is more applicable than the supervised routing metods. 2. The experimental results indicate that Smoothie can outperform other supervised routing methods on several generation tasks and can also conduct prompt selection.

Weaknesses

1. The method relies on the embedding method to measure the semantic similarity, which limits the scope of Smoothie to semantic-related tasks. The capability of LLMs should vary more on tasks such as mathematical reasoning, but these tasks depend less on semantic similarities. 2. The experimental comparisons are unclear. The chosen metrics are highly abstractive. Given the performances of each LLM are unknown, it's hard to identify whether the method successfully routes to the best or slightly better model than the baselines. The presentation of the experiment section is also hard to follow.

Questions

1. (Refer to W1 and W2) Can you provide the performance statistics of the models used in the experiments? 2. Does Smoothie require more computation than the supervised methods?

Rating

5

Confidence

4

Soundness

2

Presentation

3

Contribution

3

Limitations

The limitations and broader impacts are not discussed in the paper.

Summary

This paper proposes a method for selecting LLMs' responses for generative tasks. It can essentially be viewed as a "truth inference" problem in the research community of "weak supervision" and "crowdsourcing"; unlike ordinary truth inference methods, this paper takes into account unstructured textual information and, for each sample, the method in this paper needs to pick one out of all candidate answers.

Strengths

1) First, the problem investigated in this paper---selecting LLMs' responses for generative tasks---is very practical and interesting. 2) The proposed method is generally very intuitive and I think it will be effective.

Weaknesses

Main concerns: 1) First, authors should summarize the most relevant works in the main text of the paper. For this current version, I need to scrutinize the most relevant works in the appendix. 2) Related to the above point, there are already works that focus on the task that this paper addresses, e.g. "An error consistency based approach to answer aggregation in open-ended crowdsourcing". (Although this paper focuses on crowdsourcing workers rather than LLMs.) Therefore, it would be advisable for the authors to research the relevant literature in more depth and to consider them in experiments as comparison methods. 3) The core theoretical part of this paper is applied to the content in the existing work [72]. In the main text, it is necessary to state the technical and theoretical differences with the existing work [72] more clearly and in detail. Some other concerns: 1) The meaning of some symbols is not explained, e.g., line 109. 2) In this paper, a graphical model is presented, then a graphical representation of this graphical model should be shown. 3) Typos. E.g., "x(Section 4.2)" in line 134, "$\theta_i(x)$s" in line 137. 4) The quality of the images (e.g. sharpness) can be further improved. Also, in Figure 2, the name of the proposed method should be shown in capital letters. 5) The tables are not self-explanatory enough, e.g. the metric of interest is not shown in Table 1. 6) Some of the presentations are confusing, such as lines 609 and 610.

Questions

Please refer to the "main concerns" above.

Rating

5

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

Yes.

Summary

- This work proposes a method called SMOOTHIE, which can route label-free test examples to LLMs. Specifically, - it employs a latent variable model and Gaussian distribution for efficient quality score estimation and uses LLM outputs to estimate generator quality. - it estimates specific to each test sample using nearest neighbors and routes samples to LLMs with highest quality scores. - Empirical results show that: 1. SMOOTHIE's learned quality weights correlated with actual LLM performance. 2. In mixed-tasks datasets SMOOTHIE is able to route different samples to different LLMs which boosts the performance. 3. SMOOTHIE can be used for prompt-selection.

Strengths

- With the recent advances in LLM research, how to choose the best LLM/ how to select the best prompt for different tasks is an interesting topic and this work proposes an approach to deal with this problem. - The empirical results show that SMOOTHIE does help increase the performance for a dataset with mixed-tasks for different embedding models and different neighborhood sizes chosen.

Weaknesses

- In figure 3. there is no label on x-axis - typo: in Section 5.2, line 263 and 264, the "SMOOTHIE-independent" be SMOOTHIEGLOBAL instead.

Questions

- In the algorithm, it is not quite clear how to select indices j and k different from i. Is it just random selection? - How do you compare SMOOTHIE with Minimum Bayes method, where you choose the output from an LLM that aligns the most with all others?

Rating

6

Confidence

3

Soundness

3

Presentation

2

Contribution

3

Limitations

- The authors did not address the limitations - There is no negative societal impact

Summary

This paper presents a model for routing an input to an LLM from a pool of LLMs. The aim is to estimate which LLM will produce highest quality generation without using any labeled training data in estimating the routing model. Instead, the approach relies on weak supervision to learn the parameters of a Gaussian graphical model. The technique is evaluated by examining the model's ability to pick the oracle best LLM from the pool on various tasks; then by evaluating it's end-to-end routing performance compared to baselines (some of which have access to labeled data); and finally by using the same technique to pick amongst a set of LLM+prompt pairs for various tasks in order to effectively do prompt selection.

Strengths

- The paper makes creative use of the Gaussian graphical model from Shin et al. (2022) by using a separate embedding model (SentenceBERT) to produce an embedding representation of each {input, generation} pair from an LLM. The graphical model then provides a joint distribution over those embeddings plus the latent embedding of the {input, true output}. Inference in the model is straightforward and efficient and allows for both (1) training from unlabeled data and (2) estimation of the LLM scores conditioned on the observed input. Overall, the technique provides a simple, elegant application of Shin et al. (2022)'s model and could be easily replicated. The approach is clearly described. - The results on correlation of the graphical model quality estimates with the oracle best LLM are fairly high. - The end-to-end routing evaluation considers a breadth of tasks including both classification and generation problems. Overall the results tend to be on par or better than strong baselines, including those that use labeled training data for model selection. - Finally, the application to prompt selection is a nice addition demonstrating the applicability of the approach beyond the obvious routing application. - Each experiment considers a variety of models (as well as tasks) and most experiments include multiple pools (e.g. a 3B and 7B pool) to showcase that effects across different familys of LLMs.

Weaknesses

- The paper has no discussion of efficiency, but this seems like an important point to address. This has important implications for the motivation: if one has the capacity to run a pool of small models, why wouldn't one instead run a single larger model. In this way, the paper should really address the tradeoff of accurracy/ROUGE with # of parameters (or even more realistically runtime). - The paper does very little to discuss its limitations. There are a few comments peppered throughout. - The paper does not mention anything about which LLMs are being picked. For example, is the job of the router trivialized by certain pools of models? For example, one could imagine a setting in which the router simply identifies and then always picks the best model. The paper does no analysis of this.

Questions

- How would address the efficiency concerns mentioned in the weaknesses section? - Did you observe anything about the behavior of the routing across different tasks?

Rating

7

Confidence

4

Soundness

3

Presentation

4

Contribution

3

Limitations

Yes

Reply to rebuttal

Thanks for the thorough response! It'll be great to have these in the next version of the paper.