The Larger the Better? Improved LLM Code-Generation via Budget Reallocation

Summary

The paper compares large and smaller LMs for the code generation under the same computation budget (FLOPs or wall-time). There are two setups in the paper depending on the availability of unit tests. When the unit tests are available, generating more outputs using smaller LLMs increases the likelihood of getting a correct answer. When the unit tests are unavailable, greedy decoding with large LMs seems to be a good choice. The writing of the paper is clear and self-contained. As someone who is not familiar with code generation, I find the paper is easy to follow. The idea of turning the evaluation from pass@k to pass@flop and pass@time is simple but well-motivated given the current methodology of evaluation with pass@k. The paper executes the simple idea well.

Rating

5

Confidence

2

Ethics flag

1

Reasons to accept

- The paper could be interesting for code generation community - The work is incremental but solid

Reasons to reject

- While we see the relative improvement of smaller LMs in pass@flop and pass@time, the absolute value of pass@flop (and also pass@k) seems to indicate that current Code Llama is far from usable. Thus I felt that the recommendation of using smaller LLMs to generate multiple outputs is not as critical as improving code generation LLMs itself. - The paper mentioned the common practice of scaling LLMs as a sole drive for improving performance. This is not true. In the past years the community has advocated for better pretraining/fine-tuning data. Thus I feel that the research in the paper is not far ahead looking. - In practice, there are multiple ways to improve generation speed/quality. For example, applying speculative decoding to LLM or distilling large ones to smaller ones. Those common practices are not studied in the paper thus it makes the paper have less practical value.

Summary

The paper investigates whether smaller LLMs can outperform or match the performance of larger models when constrained by the same computational budget. The authors present a approach that utilizes smaller models multiple times within a given budget and selects the best output using a ranking method. This study is situated within the broader discourse on the efficiency of model scaling and the practical constraints of deploying large LLMs.

Rating

5

Confidence

3

Ethics flag

1

Reasons to accept

The paper reveals that smaller LLMs can outperform or match the performance of larger models when constrained by the same computational budget. The findings have practical implications, especially for applications with limited computational resources.

Reasons to reject

1. In practical applications, developers aim to reduce computational costs while enhancing performance. However, the paper only presents results under a constant computational budget. It would be beneficial for the study to explore ways to improve performance across various computational budgets. 2. The reliability of using large language models as rankers, as proposed in section 5.2, is questionable. The authors should conduct further analysis on this aspect. 3. In the related work section, the authors should also discuss the direction for enhancing generation speed and quality with smaller models (such as distillation).

Summary

This paper studies an interesting questions of given the same budget of compute or wall-time, will the smaller models outperform larger models given their lower inference cost. Experiments are conducted with the CodeLLaMA models ranging from 7B to 70B, on three function-level code generation benchmarks, HumanEval, MBPP and APPS. Results show that when test cases are available, smaller CodeLLaMA models consistently outperforms larger models. However, when test cases are not available, larger models still perform better even when likelihood-based ranking methods are used.

Rating

7

Confidence

5

Ethics flag

1

Reasons to accept

S1. The idea of this paper is very interesting, and the results potentially have great impact on how the practitioners would choose models for their specific use cases; S2. The experiments are nicely designed, the metrics as pass_flop@k and pass_time@k make a lot of sense to me; S3. The flow of this paper is quite smooth, which makes it very easy to follow

Reasons to reject

W1. The major weakness of this paper is that only one set of models (i.e., CodeLLaMA) is studied in this work, which makes it unclear whether the same conclusions would generalize to other code-specific models (e.g., StarCoder) or general purpose-models (e.g., LLaMA); W2. I found it quite strange that all of the results are reported in the form of figures, and there are no concrete numbers in such figures. For reproducibility and comparison purposes, I would suggest at least putting the concrete numbers in the appendix; W3. There are some key experiment details that are missing, see Q5 below.

Questions to authors

Q1. Do you plan to add the results for at least another set of models, either code-specific (e.g., StarCoder) or general-purpose (e.g., LLaMA)? Q2. Have you tried other reranking-based methods for code, such as MBR-Exec (Shi et al., 2022) and LEVER (Ni et al., 2023)? Also note that there are a couple of other heuristic-based reranking methods described in Ni et al., 2023. If not, can you comment on what the potential performance implications will be? Q3. Can you comment on how methods like quantization and flash-attention would affect the FLOPs and wall-time calculated in Table 1? Q4. For larger models, have different model parallelism methods been tried to ensure that maximal efficiency is achieved given the same compute infrastructure? Q5. How are the tests being used for the results reported in Section 4? For example, for MBPP there are 3 test cases, do you use all three to filter the incorrect samples, or just 1?

Summary

The paper explores the hypothesis of whether larger models are better than smaller models even in limited computational budget scenarios. In other words, the question the paper explores is: can we generate more outputs from a smaller model so that it matches the performance of bigger models under a certain computational budget? The analysis focuses on the task of code generation. The models are large language models (LLMs) of different sizes and the smallest size is 7B parameters while the largest is 70B parameters. The results of the paper show that repeated use of smaller models can yield consistent improvements with gains of up to 15% across different tasks. In scenarios in which a candidate generation needs to be ranked and then selected as the best one, the smaller model fall short of the performance of a single greedy decoding output from a larger model. It is unclear from the paper the reason for this result.

Rating

6

Confidence

4

Ethics flag

1

Reasons to accept

The paper is easy to follow, well written and well organized. The experimental settings seem robust and well designed. The paper explores an interesting and important problem that is assessing whether we actually need to use the largest model available always, under every computational budget condition. This is useful for the community that works on building such models and applications with them so that they make informed decisions when designing model architectures and approaches that rely on them.

Reasons to reject

Mainly related to the generalization of the results across different generation tasks or all types of tasks. The paper focuses solely on one task, code generation, for which there is a big interest in the software engineering community. However, it is difficult to generalize the conclusions obtained for code generation for every other generation task. The main reason is because the evaluation metrics change according to the task. It is not completely clear and focused on the paper why the ranked-selection-based task (for code generation cases without unit tests) do not work well for smaller models. Previous work [1], [2] explored re-ranking approaches to other NLP problems such as machine translation in which the the ranker is an LLM trained with human-labeled supervision about the translation quality of translated outputs. In this scenario, re-ranking with this specialized ranker usually works well but no analysis of different LLMs sizes has been done. Perhaps this is an interesting research direction that complements the work here. Overall the paper could use from experiments on other tasks besides code generation to be able to claim that smaller models are better than larger ones. [1] https://arxiv.org/abs/2205.00978 [2] https://aclanthology.org/2023.emnlp-main.733/

Thanks for the response

I would like to thank the authors for the response. I think all my concerns are adequately addressed, adding the results for LLaMA-3 will definitely make the paper stronger to show that the conclusions generalize to non-code-specific LLMs. Regarding W2 and Q3+Q4, please include these discussions in the next version of the paper as I think those would make the paper more holistic. I'm increasing my score based on the response, good luck!