Distributed Inference and Fine-tuning of Large Language Models Over The Internet

Summary

The paper proposed cost-efficient inference and fine-tuning methods for LLMs on geodistributed devices in a consumer-grade network. The motivation is that, by pooling together idle compute resources of multiple research groups and volunteers, we could make LLM research and applications accessible to broader communities. Technically. this paper comes up with an algorithm to address two challenges: 1) how to conduct the computing reliably if any device can disconnect abruptly; 2) how to to partition LLMs between devices with uneven hardware. According to their simulations and real-world experiments, the proposed method can outperforms other approaches to running inference on consumer-grade hardware.

Strengths

- The motivation of this paper is realistic. As we all know that playing with LLM is costly regarding the computing resources. Using the idle resources to do LLM inference and fine-tuning is socially and environmentally good. - The idea of this paper is clear and practical. - The paper applied multiple optimizations from different dimensions regarding the training/inference of LLM under low-resource, e.g., quantization both weights and activations between pipeline stages, efficient fine-tuning and so on. Although each of these methods are not really new, it is still inspiring to put them altogether and show that they work well. - The experiments are reasonable, especially the real-world setup.

Weaknesses

1. How to do fine-tuning under the proposed setting is not that clear, although the authors wrote one paragraph to explain the fine-tune part. Inference is clear and relatively simple, but how to recover from the failure during the training is not described and verified. 2. The communication cost from the tensor parallel is kind of missing. It seems that authors assume each server/client is able to hold a pipeline stage. However, in a more realistic scenario, each stage is further divided into multiple parts and one server/client could only hold a part of a stage. Then the intensive communication of tensor parallel may dominate the inference speed, because usually the tensor parallel only happen within one server, instead of distributed devices. 3. In Table 1, why the performance of Cache-less is irrelevant to the failure rate?

Questions

Please refer to the Weaknesses

Rating

5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

The potential limitations include privacy of data processed by outside peers, as well as broader impact of making LLMs more accessible.

Summary

The paper discuss about an important application problem of distributed inference for large language models. Given the size and inference requirement for large language models, and the constraint of hardware resources, the authors put forward utilizing idle GPUs in network to sped up the inference, providing detailed algorithm implementation and real world experimentation. Overall the work is solid and the presentation is good, and the application has important real world use cases. I have a few minor comments below.

Strengths

1. real world problem of focus - the target problem to be resolved is important 2. solid experimentation - across continents experiment to demonstrate the distributed accelerator system's performance is amazing 3. detailed implementation - algorithm detail, result comparison, and analysis are solid

Weaknesses

Overall I feel given the topic of distributed inference for LLM, the work is solid and clear. Some aspects to improve the work includes 1. server utilization discussion - the paper lacks coverage regarding the resource utilization of different GPU servers, given the distributed and heterogeneous computing setting 2. distributed accelerator infra requirement for LLMs - depending on different LLMs, there should be some basic requirement on the hardware. For example TPU is not covered, high-end CPUs are also not explored 3. expansion to LLM training - this is not the weakness of work, but would be great if the direction can be explored 4. consumer-grade network constraint - similar to 2, there are other real world constraints given the topic for the "consumer-grade network". To push the work beyond lab setting, network constraints like different firewalls, fault-tolerance mechanism considering failure rate beyond 1% (the upper bound of experimentation setting), etc

Questions

see above comments

Rating

7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.

Confidence

4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

see above comments

Summary

The objective of this study is to facilitate the operation of Large Language Models (LLMs) using commodity hardware over the internet. However, such hardware can often be characterized by high unreliability and latency issues in networks. To mitigate these challenges, the paper introduces a dual attention caches method that backs up intermediate results and supports device failure recovery. Additionally, the authors have developed a decentralized load-balancing algorithm to optimally assign transformer blocks to each server in a bid to maximize the system's overall throughput. The authors have provided a robust implementation of the proposed system and demonstrated impressive performance on the largest publicly available open-source LLM.

Strengths

* Offloading parameters of LLM to remote devices instead of local local storage (e.g. SSD) make senses. Although the former could have higher bandwidth, the IO amount could be much higher than the later. Emprical results also support this analysis. * The analysis on inference chanllenges is interesting. For example, the communication cost, past token storage. It helps the community understand the chanllenges of serving a large language model with billions of parameters. * client/server caching, shorest path routing, and automatic load balancing improve robustness, effiiency and soundness of the system.

Weaknesses

* I found that Table 2 is a bit vague to follow. Please eborate the metrics steps/sec and tokens/sec per user. For example, the difference between step and tokens and between clients and users.

Questions

* For the comparsion between distributed inference and local offloading in table 2, does the local offloading also use the qunatized version of BLOOM? A clarification on this will help understand where the improvement of distributed inference comes from.

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

* Precise and explicit definitions of client and server are missing. * Missing references on offloading inference compute across multiple devices [1,2, 3]. Discussing these would provide a more comprehensive context and enhance the depth of analysis. [1] Kang, Yiping, et al. "Neurosurgeon: Collaborative intelligence between the cloud and mobile edge." ACM SIGARCH Computer Architecture News 45.1 (2017): 615-629. [2] Matsubara, Yoshitomo, et al. "Head network distillation: Splitting distilled deep neural networks for resource-constrained edge computing systems." IEEE Access 8 (2020): 212177-212193. [3] Dong, Xin, et al. "Splitnets: Designing neural architectures for efficient distributed computing on head-mounted systems." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.

Summary

This paper presents a system designed for decentralized inference and fine-tuning of large language models over distributed hardware, which allows users to efficiently run LLMs without requiring high-end hardware. The system leverages pipeline-based model parallelism, distributing model layers across nodes. Additionally, this work proposes fault-tolerant inference algorithms and load-balancing protocols to enable dynamic deployment of models. During the evaluation, experiments were conducted using the BLOOM-176B model, which demonstrated a 10x improvement in performance compared to the offloading method.

Strengths

The writing of this paper is fluent, with clear and logical reasoning, and the proposed solution is well-aligned with the requirements； The design is consistent with current trends and can effectively address the problem, making it highly practical； This paper presents a comprehensive system, with reasonable comparisons made against upper bounds and benchmarks.

Weaknesses

The innovative aspects of the paper are not sufficiently elaborated upon. For example, I would like to know if there are any outstanding advantages compared to the latest research such as DeepSpeed, aside from differences in application scenarios. Has there been any comparison of similar metrics? The results are somewhat unsatisfactory. Caching with restarts appears to be quite competitive, and the algorithm proposed in the paper performs better only under high failure rates. However, for short sequences, Cache-less inference performs better under high failure rate conditions (which may be due to some inherent communication issues in system). It would be helpful to compare several lengths or plot curves to more clearly show the trend and find the optimal point. Although the focus of the research is on the inference process, it would be best to systematically elaborate on the experimental results and methods related to fine-tuning for the sake of experimental completeness. A minor suggestion is to cite Algorithms 2/3 more specifically in the text to make the writing clearer. I am not particularly familiar with distributed learning, so please forgive me if I make any inaccuracies in my statements.

Questions

The paper mentions that autoregressive LLM inference cannot be performed with a single pass through the model, leading to a higher system complexity. Do the proposed algorithms still have advantages for models that are not autoregressive? Was this the primary design point of the system, or is it also applicable to other models? During the evaluation of offloading, how many GPUs were used for benchmarking? Is there a strict basis for the best-case scenario?

Rating

6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.

Confidence

3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.

Soundness

3 good

Presentation

3 good

Contribution

3 good

Limitations

(1)To better evaluate the system, it would be beneficial to extend the experiments to include more models. (2)Data privacy is a concern, as multiple clients may contribute to data misuse. It would be helpful to propose some solutions to address this issue. (3)Furthermore, the innovative aspects of the paper could be further clarified.