Cheaply Estimating Inference Efficiency Metrics for Autoregressive Transformer Models

Summary

The authours proposed a metric called `idealized runtime` to evaluate the inference efficiency among different LLMs as if they were run on the same standard hardware/software system. The main contributions can be summarized as following: * the `idealized runtime` for a specific query with prompt length `p` and output length `o` on a target LLM can be collected by running the LLM on standard hardware/software (e.g., A100 and Megatron containers) **if target LLM's architecture is known/opensourced** * the `idealized runtime` of transformer models can be modeled as a function linear to `o` and piecewise linear to `p` **if context window size is much smaller than embedding size of the model**. So with a few queries with different `p` and `o`, the coefficients of this function can be extracted by fitting the runtime of these queries to the function. * with the fitted function, `idealized runtime` of queries can be estimated without running them exhaustively, so that the capacity of model (e.g., accuracy on a dataset/benchmark) can be compared against the pre-calculated/estimated `idealized runtime` of all the queries in the dataset to tradeoff/review scaling effect of LLMs. * for closed models with api access, the authors proposed an alternative `denoised runtime` to approximate their idealized runtime.

Strengths

* The authors pointed out the scaling law of LLMs should not just focus on accuracy vs FLOPs/Model size but accuracy vs inference latency/cost/power as well, which is a valuable point in LLM production. * The authors proposed a simple way to estimate the idealized runtime of queries on standard hardware/software systems given prompt length and (projected) output length.

Weaknesses

* The inference latency of a well known (open sourced) LLM architecture can be easily estimated by total ops of the model (e.g., those calculated in line 105/115) and divided by hardware effective (typically around 50% for LLMs on A100, https://arxiv.org/pdf/2104.04473.pdf or BLOOM paper) FLOP/s. The authors only mentioned about the drawback of this proxy briefly in line 300-308, without evidences based on target LLMs. Thus the significance of approximating these latencies via fitting instead of analytical calculation is quesitonable. * The work makes too many assumptions that devalue its practice, 1. context length << embedding size. This is true when the common context lenght is only 2K, but new development on LLMs have pushed them to 32K(GPT4)/64K (MPT)/100K(Claude), so this assumption and therefore the linear function approximation is no longer valid. With long context/prompt length, the analytical equation is probably more accurate. 2. the `idealized runtime` is collected on standard Megatron container assuming model architectures are known. This limits the effectiveness of the work to open sourced models with efficient (distributed) implementations. 3. the work assumes the pretrained/finetuned model is the final service/inference model. However in practice, the service model can be quantized/sparsified/distilled, so apple-to-apple inference latency comparision on pretrained models doesn't direclty translate to service model and the gap is typically huge, e.g., as shown by FIG3, the variance in idealized-runtime vs denoised-runtime is significant. * The presentation of the work is not clear and sometimes repetitive/redudant. 1. using FIG3 again as example, what's the diff of the two subplots in (a)/(b), it is not clarified and seems just different scales in axis. And as mentioned above, this FIG suggests for a couple models, the denoised-runtime is smaller than idealized-runtime, which is not a good evidence that idealized-runtime is a good approximation of roofline runtime. 2. A lot of the equations are just rewrite/modifications to a simple linear function, and probably can be omitted or summarized in a table. * Some of the conclusions are drawn without evidence, e.g., in line 257, the authors claimed "This suggests that scale alone does not predict model capabilities." Scaling law should be studied for the same model by varying model/dataset size. Comparing different model families with drastically different in pretraining dataset size/quality can not lead to any conclusion over model capacity over model size or inference latency. * Overall the work is oversimplified by the assumptions above to a linear regression problem, which can be analytically calculated (and is commonly adopted, e.g., like Palm2 tech report) and becomes trivial to the audience of NeurIPS.

Questions

Basically 1st point described in weakness. How far off is simple FLOPs estimated latency from the estimated `idealized runtime`, can you do a quantative study to support the necessity of doing this linear function fitting/approximation? And if you found the effective FLOP/s of an LLM is much lower/higher than 50%, can you study whether it is intrinsicaly due to the model (e..g, many non-dense ops) or the implementation.

Rating

2: Strong Reject: For instance, a paper with major technical flaws, and/or poor evaluation, limited impact, poor reproducibility and mostly unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

1 poor

Presentation

2 fair

Contribution

1 poor

Limitations

The limitations are mainly the assumptions the authors made, as mentioned in weakness. The authors either didn't discuss them or without referring to SOTA LLM developments.

Summary

The authors proposed idealized runtime, a metric for measuring inference efficiency in LLMs, which measures the performance of models as if they were executed on a given hardware and software platform. Idealized runtime efficiently can be extended to estimate the idealized energy and dollar cost. The metric also takes into account the amount of hardware required to perform inference on a given model. Using these metrics, the authors compare ten state-of-the-art LLMs to demonstrate inference efficiency-capability tradeoffs. Overall, the authors aim to provide a more accurate and fair comparison of LLMs across different providers and to shed light on the tradeoffs between inference efficiency and capability.

Strengths

1. The paper presentation is clear and the metric proposed is intuitive to understand. 2. The empirical study is thorough and examines the LLM inference efficiency from various perspectives. 3. The empirical findings demonstrate that the proposed metric can largely capture the efficiency-capability tradeoff in LLM inference.

Weaknesses

1. The models on different tasks and metrics exhibit vastly different Pareto Frontiers (Figure 9). The cause of this variance hasn't been fully explored and explained in the work. Can these datasets/tasks properly evaluate the capabilities of the LLMs or is the variance due to something else?

Questions

I have raised my concerns in the previous section.

Rating

8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

4 excellent

Presentation

4 excellent

Contribution

4 excellent

Limitations

The authors may further discuss the potential negative societal impact of their work.

Summary

This paper aims to better understand the tradeoff between the inference efficiency and capability of LLM. It comes up with a metric called “idealized runtime” to fairly evaluate the inference efficiency of several popular LLMs, which is designed to factor out the irrelevant noises such as hardware, code stack, network latency and so on. With this metric, the authors analyised the efficiency and capability of LLMs and got some inpsiring insights.

Strengths

First of all, I found the motivation of this paper very realistic and helpful to the community. We have been seeing lots of efforts made by the industry and academia to reduce the cost of the inference of LLMs, but how to compare those models from different parties fairly is yet to be figured out. This paper would be an inspiring exploration to this end. Besides, the analysis regarding the LLM inference is thorough and clear.

Weaknesses

- Some technical details are kind of missing, which I will specify in the Question section. - The concept of “idealized runtime” could be a bit unpractical because it requires one to implement and deploy the LLM on a very specific hardware and software setting, which may involve quite amount of proper engineering efforts.

Questions

- As I understand, raw runtime and denoised runtime are derived from the API of model providers, and the idealized runtime is derived from the local cluster. Then how to interpret the impact of batch-size? As we know, the batchsize plays a quite import role in the padding, scheduling and I/O overhead during the inference, which could lead to very difference runtime. Since the batch size behind the API is unknown, how do we know if the differences between raw/denoised time and idealized time is actually simply dominated by the batch-size, instead of hardware and software? - I wonder how to get the denoised runtime specifically? In the paper it says “we can run multiple trials in the profiling step and perform linear regression using the *minimum* obtained runtime across trials for each p and o”. So I guess it means you sent the same queries to a API n times, and chose the minimum runtime as denoised time? But how to do this sampling properly is important, for example, you may have quite different response time between morning and evening because of the traffic volume. And again, the impact of batch-size cannot be denoised here. - In Figure3, why there are two figures in each subfigure? In the left figure of Fig3(b), why the y-intercept is not zero? - Regarding the idealized runtime, it will be helpful if the authors list the detailed specifications of hardware and software setup, such as, CUDA version, Megatron version, tensor / pipeline parallel degrees, inter/intra node connections of GPUs, inference batch size. - From Fig4, we can learn that the idealized time of different models actually aligns relatively well with the model size and FLOPS (except the little difference of BLOOM and YaLM), whereas the raw and denoised runtime is way more different than model size. How can we interpret this observation? Could we just use the model size and number of GPUs needed to roughly estimate the idealized time? - Following the method described in section 3.4, we could estimate the linear regression function of prompting and generating time from an API, which is interesting. But in the experiment section, we only see the raw/denoised time in total, instead of breaking into prompting and generating time. I see a table in the Appendix about this data, but it would be nice to present a bit of them in the main paper as well. Otherwise the readers may wonder why do we need Section 3.4 if all the experiments are just end-2-end runtime.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The potential social impact is not discussed in the main paper.

Summary

"This paper introduces a range of metrics designed to compare the tradeoffs between inference efficiency and capability of autoaggressive Language Model (LLM) based on Transformers. Alongside the raw run-time metric, which can be influenced by infrastructure optimization and performance variance, the paper proposes additional metrics, namely idealized runtime, denoised runtime, idealized dollar cost, and idealized energy cost. These metrics enable a comprehensive comparison of various LLMs, even when they are served through different black-box APIs and hardware/software implementations.

Strengths

* Originality: The paper introduces a novel analytical approach to estimate the inference efficiency of autoregressive Transformer models, overcoming challenges posed by models accessible only through black-box APIs and running on diverse hardware and software. This distinct proposal stands apart from previous analytical models designed for non LLMs. Moreover, it addresses the limitations associated with other proxies, such as model size and FLOPs. * Quality and clarity: The proposal is presented with a rigorous analytical description and validated through empirical verification. The paper's structure is well-organized, and the content is conveyed in a clear and concise manner. * Significance: The metrics, encompassing both the newly proposed and conventional ones, are effectively employed to compare the tradeoff between capability and efficiency. This comparison leads to several noteworthy insights, elaborated upon in Section 5.4.

Weaknesses

There are a few minor formatting issues that require attention. For instance, in line 153, the notation of $o^2$ is confusing. The "2" is intended to refer to the footer, but it appears as if it denotes an exponent of 2.

Questions

* In Figure 3, the denoised runtimes of OpenAI models are often lower than the idealized runtime. Could you provide further explanation for this phenomenon? Is it possibly related to the presence of better-optimized hardware and software infrastructure? * Can you clarify which variables are being compared in Table 2? Are the comparisons made between Raw runtime and idealized runtime?

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

As mentioned in Section 3.1.1, the analytical model does not account for models with very large context windows (> 10,000); however, it is worth noting that these models are now becoming available.

Summary

LLMs are dominating the NLP space at the moment, but efficient inference run-time and cost estimations have not been explicitly defined and tested. Raw run-times are not particularly useful across multiple services providing LLMs, as there is typically high variance overhead via the presence of black box prompt APIs. This paper proposes a few ways of estimating inference costs that attempt to provide an upper bound to the costs incurred by model computation. These metrics are demonstrated to be generally accurate and are cheap enough to be easily implemented, and the authors attempt to use them to generate new insights concerning various LLMs and generally large models across multiple services.

Strengths

1. Very relevant, as LLMs are dominating conversations in the NLP space. Paper as a whole does a good job of making a case for the necessity of their work. Methodologies to estimate a separation of overhead costs from core model computation are inherently useful. Most, if not all, assumptions are fairly setup and seem correct. 2. The paper is very well-written. The problem setup is executed well and the initially proposed solution is very easy to follow. Explanations are mostly succinct, and most sections read as cogent. 3. The results look to be generally correct and explanations related to them are succinct. The results are made more relevant by the scale of the experiments, rendering the paper somewhat unique. The applied denoising factor estimation technique seems to roughly work in providing an upper bound for expected model computation time.

Weaknesses

1. Some areas of the paper are a bit overexplained for this venue. Readers and reviewers of NeurIPS should, for example, be very familiar with autoregressive frameworks and how they generally work, or the forward pass design of generative AI in general. 2. Several early conclusions are mostly obvious. A lot of time is spent on specifying a procedure to estimate the number of floating point operations (and subsequently runtime based on throughput), but that isn't really all that interesting or new. For example, LLMs having roughly linear costs with respect to the size of their input (their quadratic attention costs are dwarfed) is probably well understood by your audience. Empirical proof related to this likely shouldn't be core to the paper. 3. Unclear if particularly new insights were generated from the results. There were largely no clear winners in cost-capability tradeoff analyses, there is no comparison to other evaluation techniques in the core body of the paper, and insights into when models fall or do not fall on the Pareto frontier of a given task do not seem initially useful.

Questions

1. Some noticeable run-to-run variance was observed for certain models, but I did not spot exactly how it was dealt with beyond the selection of a minimum time. Could you elaborate on this point, especially for AI21/J1-Grande v1? 2. What would comparable exhaustive profiling cost a designer/user compared to your method? Can you provide some sort of comparison? Are there specific techniques you can compare your method to? Some materials from the appendix are mentioned, but this seems like it should be core content and included in an expanded Section 5.3.

Rating

4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

n/a

Summary

This paper presents a new metric that attempts to more fairly evaluate machine learning model inference runtime performance cost, which the authors refer to as idealized runtime. The authors argue that the black-box nature of many large language models (LLMs) does not necessarily provide accurate estimates of inference cost using raw runtime results alone, due to several external factors (e.g., hardware used, compute resource contention, etc.). Thus, a goal of idealized runtime is to create a fair(er) inference evaluation metric for LLMs. The authors analyze 10 state-of-the-art LLMs using their idealized runtime metric and discuss the efficiency tradeoffs between them.

Strengths

- The paper is well-written - The research topic of creating a somewhat universal metric to measure spatial and temporal overhead of LLMs on inference is relevant and timely - The authors’ intuitions on focusing on compute operations (e.g., floating point instructions) and performing an inference runtime calculation that is not overtly computationally intense, is a good foundations for such research

Weaknesses

While I agree with the authors that this topic is certainly worth deeply investigating, I have too many open concerns to recommend it for acceptance (at this time). Below I list two areas that I believe would need to be addressed before can be reasonably considered for publication. - floating point operation overhead (a critical paper weakness): in Section 3.1.1 the authors discuss some of the concrete details used for their idealized runtime equation. Of note is the focus on deriving expressions “for the number of floating-point operations required for each of the two steps, and then use these to derive expressions for runtime.” While the number of floating point operations required to perform a single forward pass of inference is quite likely to an important datum for estimating inference runtime overhead, it is (in my opinion) just the first of many (perhaps dozens?) of factors that may be required to make an accurate estimation of inference overhead. There are several reasons for this, I list a couple below (the below list is not exhaustive). (1) Not all floating point operations are spatially or temporally equivalent (e.g., a floating point ceiling or floor operation is notably less expensive than even a mathematical divide operation due to the execution “trap” that must exist for the divide, but not for the floor/ceil). Moreover, some floating point operations are orders of magnitude different in overheads (e.g., an FP divide is likely 10x-100x faster than a gather-scatter vector FP operation). To attempt to quantify all FP ops in a single computational class is logically unsound. (2) The spatial and computational complexity of FP operations between different hardware compute classes varies significantly (often times more than 10x-100x). For example, GPUs can handle up to their channel size limit of FP data and continue to emit a fixed computation time for equivalent class FP operations. This is due to GPUs inherent SIMD properties. CPUs, on the other hand, emit approximately linearly increasing FP operation overhead as FP operation count increases. On the other hand, if the GPU channel size is exceeded even by a single datum, the GPU execution time may increase by 2x or more because it will generally require a second issuance of GPU block execution. Yet, CPUs only suffer roughly additive execution overhead when additional datum are added for FP operations. This means there is generally an inversely proportional runtime cost associated with FP operations and data between CPUs and GPUs. The paper seems to only focus on GPUs, which, in commercial practice, tends not to be representative (or at a minimum highly specialized) of which compute is used for inference. - References: of the 50 or so references included by the authors, it appears that only around 10 of them are from peer-reviewed publications. The other 40 or so are from arxiv and websites. Moreover, the arxiv references are not necessarily recent (some date back over 5 years ago). The lack of peer-reviewed citation raises questions about the validity of substantiation as well as the authors’ understanding of the domain. A core problem with citing non-peer-reviewed published work is the lack of a fixed written artifact. Arxiv papers can have many versions as can website/webpages (which version are the authors referring to and how would a reviewer know this a priori?). I strongly encourage the authors to perform a more rigorous literature review and to substantiate their claims principally through peer-reviewed scientific research. It is a tall order to expect a reviewer to personally review / validate every non-peer-reviewed research paper to help verify its soundness and thus its appropriateness when used to substantiate the authors claims. Yet, not doing so would seem to imply that reviewers would need to implicitly “trust” the quality of such non-peer-reviewed artifacts. This seems antithetical to the evidence-based scientific research that is published at NeurIPS.

Questions

None.

Rating

2: Strong Reject: For instance, a paper with major technical flaws, and/or poor evaluation, limited impact, poor reproducibility and mostly unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

1 poor

Presentation

2 fair

Contribution

1 poor

Limitations

The authors seem to only focus on inference runtime overhead associated with GPUs (and a single class of GPU, as a reference point, Nvidia A100) and no other form of hardware compute. However, many other types of hardware are used to perform inference. Some of them are: CPUs, FPGAs, TPUs, CGRAs, etc. Moreover, each of these hardware classes have subclasses within that result in different temporal overhead and power footprint costs. None of these factors seem to be considered in the authors runtime equation.