Addressing the challenge of effectively processing long contexts has become a critical issue for Large Language Models (LLMs). Two common strategies have emerged: 1) reducing the input length, such as retrieving relevant chunks by Retrieval-Augmented Generation (RAG), and 2) expanding the context window limit of LLMs. However, both strategies have drawbacks: input reduction has no guarantee of covering the part with needed information, while window extension struggles with focusing on the pertinent information for solving the task. To mitigate these limitations, we propose Chain-of-Agents (CoA), a novel framework that harnesses multi-agent collaboration through natural language to enable information aggregation and context reasoning across various LLMs over long-context tasks. CoA consists of multiple worker agents who sequentially communicate to handle different segmented portions of the text, followed by a manager agent who synthesizes these contributions into a coherent final output. CoA processes the entire input by interleaving reading and reasoning, and it mitigates long context focus issues by assigning each agent a short context. We perform comprehensive evaluation of CoA on a wide range of long-context tasks in question answering, summarization, and code completion, demonstrating significant improvements by up to 10% over strong baselines of RAG, Full-Context, and multi-agent LLMs.
Paper
Similar papers
Peer review
Summary
This paper proposes a new method, "Chain-of-Agents" (CoA), to augment the long-context handling capabilities of large language models (LLMs). CoA is a framework designed to enhance the processing of long contexts by sequentially using multiple agents to handle different chunks of input text. In CoA, worker agents manage different segments sequentially and communicate their findings. These findings are then synthesized by a manager agent into a coherent final output, effectively aggregating information and reasoning across the entire context. The author conducted experiments on a wide range of long-context tasks to verify the advantages of CoA.
Strengths
1. The proposed Chain-of-Agents method is an interesting approach, and the authors have experimentally validated its effectiveness. 2. The writing and presentation of the article are great, making it easy to read and follow overall.
Weaknesses
1. My primary concern is the novelty of this submission. Breaking long texts into multiple chunks and processing them sequentially is a well-established practice in both long-content processing and generation. For example, in RecurrentGPT [1], the authors introduced a long-short memory mechanism to store intermediate states during processing, improving the quality of long content generated by LLMs. This is similar to the communication unit (CU) mechanism mentioned in this paper, but the authors do not sufficiently discuss this similarity. Additionally, previous work such as LongAgent [2] proposed segmenting the input long text into several chunks and assigning them to corresponding members. The authors lack sufficient discussion and experimental comparison on utilizing memory mechanisms or agent mechanisms for processing long information. 2. The baselines used in the author's experiments are relatively weak. While the related works section discusses some studies on long text processing, such as references [9] and [13], the implementation process only compares RAG, Vanilla, and COA. The author primarily compares their own designed merge and hierarchical methods in Agent and mechanisms for processing long texts. So, it is challenging to determine the advantages of the proposed methods over existing strong baselines in the field based on the current experiments. 3. Given the existing work on agents collaborating to process multiple chunks, the authors could enhance the novelty of their submission by delving deeper based on the discussion in Section 5.6. Specifically, they could explore the most effective methods of collaboration in multi-chunk processing and agent cooperation. This could add significant technical depth to the paper. [1] Zhou, Wangchunshu, et al. "Recurrentgpt: Interactive generation of (arbitrarily) long text." arXiv preprint arXiv:2305.13304 (2023). [2] Zhao, Jun, et al. "LongAgent: Scaling Language Models to 128k Context through Multi-Agent Collaboration." arXiv preprint arXiv:2402.11550 (2024).
Questions
1. How is the chunk split? I did not find a clear description of how the chunk is processed. Could there be a risk of text being abruptly truncated in the middle? 2. As mentioned in the abstract, "input reduction has no guarantee of covering the part with needed information." I am wondering if sequential chunk processing will also encounter these challenges. Could the communication unit potentially drop necessary information if some information is only useful in the context of the following content? 3. In Appendix 1, regarding the calculation of time complexity in lines 592-595, if the model's processing length limit is k and no additional mechanisms are introduced, is the computational complexity of attention calculation O(k^2) or O(n^2)? Please educate me if I miss something important. 4. "a LLM" in line 104 -> "an LLM"
Rating
5
Confidence
5
Soundness
2
Presentation
3
Contribution
2
Limitations
The authors adequately addressed the limitations.
Reponses from Authors (Part 1)
Thanks so much for carefully reading our responses, and providing such insightful questions and thoughts! Below, we first re-state our novelty aspects more clearly and then comment on the raised concerns one by one, hoping to make our claims clearer to you and address your concerns! CoA has a unique positioning in the multi-agent LLM literature. The table below illustrates the comparison of related work and their topologies (after carefully reading the survey [1], [2], and other literature). Broadly, prior work on multi-agent LLM systems for long input tasks has a centralized design, and existing decentralized designs do not extend effectively to long context tasks. To the best of our knowledge, CoA is the first to use a centralized structure on long input tasks and is more effective than baselines such as WalkMaze and RAG. | Work | Task | Type | Communication Schema | |---------------------------------|---------------------------|---------------|-------------------| | [Chen et al., 2023d] | Multi-robot planning | Decentralized | Circle | | RoCo [Mandi et al., 2023] | Multi-robot collaboration | Decentralized | Planning Pipeline | | CoELA [Zhang et al., 2023c] | Multi-Agents cooperation | Decentralized | Memory Module | | MAD [Du et al., 2023] | Improving Factuality | Decentralized | Debate | | Reconclie [Chen et al., 2023] | Reasoning | Decentralized | Round Table | | WalkMaze [Chen et al., 2023] | Long Input | Centralized | Tree Structure | | LongAgent [Zhao et al., 2024] | Long Input | Centralized | Tree Structure | | RecurentGPT [Zhou et al., 2024] | Long Output | Decentralized | Memory Gating | | CoA (Ours) | Long Input | Decentralized | Chain of Agents | Besides, our contribution is not to claim a new topology for communication. The challenge of long dependency on long input tasks is well-known and remains under-explored (as illustrated with the sample in Part 2). Our target is to *novelly* mitigate **challenging context dependency issues** in long-context tasks particularly those requiring complex reasoning. Thus, it is better if we can find a **simpler yet effective approach**. To this end, we explored various structures (WalkMaze, Merge, Debate, Group Discussion, etc.) and found that a variant of **decentralized well-existing chain communication and this simple approach can work more effectively** than others. Our contribution is not to claim a new topology for communication but to mitigate a challenging issue (long dependency) with a simple intuitive approach (decentralized chain communication). In addition, CoA is not in conflict with existing work as it can be considered as a plugin for other multi-agent LLM systems (e.g. serve as an additional stage in LongAgent) to improve them. Specifically, previous work for long input tasks, despite showing significant improvements, usually uses disentangling algorithms such as question decomposition, to address long dependency, and then they use centralized structure to answer the disentangled questions one by one. Different from them, CoA does not disentangle the question but leaves the entire question to the agents. Although chain communication is well-introduced in the literature, to the best of our knowledge, CoA is the first work to apply decentralized communication to long-term dependency challenges - please let us know if there is any paper we are missing on this. We show the high effectiveness of CoA in mitigating agent dependency issues for long inputs and we believe it would constitute a significant value add in the multi-agent system literature. We will better clarify these novelty points in the Introduction section of our paper. Below are the detailed answers to specific points: > various communication mechanisms exist in multi-agent systems Indeed, chain communication has been introduced as a very simple topology of communication. Our contribution is not to claim a new topology for communication but to mitigate the challenge of long dependency. We bring a novel perspective to the challenge with the simple chain communication mechanisms. Also, to the best of our knowledge, CoA is the first to use a decentralized structure on long input tasks (see table above). We note that no other multi-agent method in the literature has such results of outperforming RAG for long inputs.
Reponses from Authors (Part 2)
> the central agents in centralized communication essentially function as the communication units described in your paper. Although they all accomplish the task of communication, we think these two types of communication, centralized communication (e.g., LongAgent) and decentralized communication (our CoA), differ in their approach to solving long context dependency issues. When a centralized approach faces a multi-hop question, it leverages question decomposition to disentangle the long dependency issue, while decentralized communication leverages the interleaved reading-processing. Below is an example: ``` Question: Who is the grandson of A? Source: [1],[2],[3],[4] (chunks) Evidence in each chunk: [1: A’s husband is D], [2: A’s son is B], [3: No evidence], [4: B’s son is C] ``` ``` Centralized communication (e.g., LongAgent): Round 1: Manager: Who is the son of A? Worker with [2]: It is B! Others say unknown Round 2: Manager: Who is the son of B? Worker with [4]: It is C. Others say unknown Final answer: It is C. ``` In this approach, a worker with [i] and a worker with [i+1] do not communicate. Thus, it is difficult to deal with the dependency when the answer to the question is split into the end of agent i and the start of the agent i+1. ``` Decentralized communication (Our CoA): Manager: Who is the grandson of A? Workers: [1]: A’s husband is D (topic exploration), [2]: A’s son is B (answer first hop), [3]: A’s son is B (forward previous evidence), [4]: A’s son is B, B’s son is C. Thus, A’s grandson is C. (complete reasoning) Final answer: It is C ``` This exemplifies how adjacent workers communicating in CoA would differ from the centralized one as CoA agents receive previous evidence in addition to the question itself. We want to emphasize that LongAgent and our work are solving the long context problem in different ways, and they are not in conflict with each other. They can be merged into a much stronger framework (e.g., adding our chain reference stage to mitigate context dependency issues before the stages in LongAgent). > Could you clarify how CoA addresses this issue? From my understanding, CoA does not involve the decomposition of the input question. The key aspect of CoA decentralized communication is that workers can communicate so that the question does not need to be decomposed. Our experiments highlight the effectiveness of this design, especially when question decomposition is difficult. For instance, when the passage does not mention “A’s son” directly but mentions “A’s husband is D”, and mentions “D’s son is B”, the first question in the centralized approach should be “Who is A’s husband” rather than “Who is A’s son”, which is difficult to propose. > However, this further diminishes the novelty of the proposed method. We acknowledge that chain communication is not first used in our work. But we are the first ones that use chain communication for long dependency in the **input of long context** tasks and we show its effectiveness on generic tasks. This approach is independent of RecurrentGPT, as RecurrentGPT solves **long output**. For the output, the issue they mitigate is memory and planning for story generation tasks. Combining CoA and RecurrentGPT to solve Long-Input-Long-Output generation could be promising as well and we leave that to future work. > many components of the proposal are common practices in multi-agent system design and long-content generation/processing. We truly appreciate your thoughtful summary. We understand your concerns and would like to re-emphasize the value of our contributions: our work presents a novel and straightforward solution to the long-standing challenge of long context reasoning, and our results demonstrate significant effectiveness. We are hopeful that our findings will inspire further innovation in the research community, encouraging others to integrate our simple yet powerful components into their future work in multi-agent systems. > it should be comprehensively tested as a stronger baseline (e.g. LongAgent) across all benchmarks you tested. Thanks for the suggestion. Indeed, LongAgent is an important work in this direction. We try to compare all datasets in our paper. However, they did not open source their code, making it difficult to reproduce and compare with them fully in this short-time rebuttal window. Since we found that CoA is 15% higher than that of LongAgent (97% vs. 82%) on NIAH PLUS (Figure 2 in pdf), we believe that given the strong improvement, the performance of CoA will be promising. We will include detailed comparisons across all datasets in the final version of our paper. We hope our response has addressed your concerns and clarified the contributions of our work. We welcome any further questions you may have and would be happy to provide additional clarification if needed. We deeply appreciate the time and effort you've taken to provide feedback on our work.
Thank you for the clear response, which effectively addresses my concerns regarding how the CoA distinguishes itself from existing multi-agent research. Including these discussions in the paper will enhance its quality and emphasize your contributions. I would like to raise my score to 5.
We are really glad that our responses address your concerns! Thanks so much for your thoughtful questions and insightful discussion. We will carefully include our discussion in the final version. We deeply thank the effort you made during the whole process!
Summary
The paper "Chain of Agents: Large Language Models Collaborating on Long-Context Tasks" introduces a novel framework called Chain-of-Agents (CoA) to address the challenge of effectively processing long contexts in large language models (LLMs). The CoA framework leverages multi-agent collaboration, where multiple worker agents sequentially handle different segments of the text and a manager agent synthesizes these contributions into a coherent final output. This method aims to mitigate the limitations of input reduction and context window extension strategies by enabling effective information aggregation and context reasoning across various LLMs. The framework is evaluated on a range of long-context tasks, including question answering, summarization, and code completion, demonstrating significant improvements over existing methods.
Strengths
- **Task Agnostic and Training Free**: CoA is a versatile framework that does not require task-specific training or fine-tuning, making it applicable to various long-context tasks without additional training overhead. - **Significant Performance Improvement**: The framework shows significant improvements, up to 10%, over strong baselines like Retrieval-Augmented Generation (RAG) and full-context models in multiple long-context tasks, including question answering, summarization, and code completion. - **Mitigation of Long-Context Issues**: CoA effectively addresses common issues associated with long contexts, such as the "lost in the middle" phenomenon, by assigning each agent a shorter context, thus maintaining focus on relevant information.
Weaknesses
- **Complexity in Implementation**: Implementing a multi-agent system could be more complex and resource-intensive compared to single-agent systems. - **Communication Overhead**: The sequential communication between agents might introduce latency and inefficiencies. - **Evaluation Scope**: While the paper evaluates on multiple datasets, more diverse real-world applications could further validate the robustness of CoA, like NIAH test.
Questions
- Could you test the CoA framework on smaller long-context models, such as qwen2-7b and llama3-8b, which both support an 8k context? I am curious to see how these smart, smaller models perform with CoA. - Regarding token cost, the input tokens are slightly higher than those in baseline methods. What about the output tokens? How are they related to the number of workers ($l$) and how each worker agent summarizes each chunk? - In Section 5.2, how do you handle the Full-200k method, which is the Vanilla (200k), when processing contexts over 200k length?
Rating
5
Confidence
4
Soundness
2
Presentation
2
Contribution
2
Limitations
N/A
Dear Reviewer, Since we are approaching the end of the discussion period, we are wondering if you have had the chance to review our response to your feedback. We would like to kindly inquire about the extent to which we have successfully addressed the concerns outlined in your review. We greatly value your feedback and would appreciate any further questions or comments you might have. Thank you for your time and consideration. Sincerely, All Authors
We are glad to know our responses addressed your concerns! We would also like to thank you for your thoughtful questions and insightful discussion, and we will include all our discussion results in the final version.
Summary
The paper proposed Chain-of-Agents, a multi-agent LLM collaboration framework for solving long context tasks, where multiple worker agents sequentially comprehend and communicate to handle different segmented portions of the text, and a manager agent, at last, synthesizes these contributions into a coherent final output. The paper conducted experiments on 9 long-context tasks in qa, summarization, and code completion, showing a significant performance gain over vanilla long-context LMs and RAG baselines. The paper also provides abundant analysis.
Strengths
- The proposed method is intuitive. - The experiments are comprehensive, covering 6 different LLMs as backbones and evaluated across 9 benchmarks. - The results are good. - The paper provided useful analysis for more insights. - The paper is well-written.
Weaknesses
- There is no support in the paper for the claim in L35 "inspired by human-like processing of long-context task", weakening the motivation. - Results on BookSum are missing in the main table, and no RAG baselines are provided on this benchmark. - Though the overall results are promising, I'm curious whether there is some information loss during the sequential "read and consume", and how this may affect the performance and the design choice. - For the case study in Figure 5, for worker 1, there seems to be more than one line of clues to answer the query (a.k.a other space missions such as "Voyager"), which may result in a complicated reasoning graph, how is this phenomenon dealt with the proposed method? - As also noted in the paper, there is no interactive communication between the agents, but only uni-directional and one-time. Whether the approach could be considered "communication" is debatable. - Although the time complexity, in theory, is true for Table 2, how's the actual inference speed on GPU considering the overhead of multiple times of extra prompting and decoding? - It might be helpful to also discuss the related works on communication between language agents, e.g. - Camel: Communicative agents for" mind" exploration of large language model society, NeurIPS 23 - Building Cooperative Embodied Agents Modularly with Large Language Models, ICLR 24
Questions
Please see Weaknesses
Rating
7
Confidence
4
Soundness
3
Presentation
3
Contribution
3
Limitations
As the paper also noted, the communication effectiveness and inference efficiency could be further improved.
Dear Reviewer, Since we are approaching the end of the discussion period, we are wondering if you have had the chance to review our response to your feedback. We would like to kindly inquire about the extent to which we have successfully addressed the concerns outlined in your review. We greatly value your feedback and would appreciate any further questions or comments you might have. Thank you for your time and consideration. Sincerely, All Authors
Summary
This paper is about addressing the issue of lengthy inputs when using language models. Predominant approach is RAG, but is hampered by retrieval performance. Window extension extends the architecture of the model to handle lengthy inputs, but doesn’t guarantee that the model is able to extract the relevant information from lengthy inputs. This work proposes a simple approach where text is split into multiple chunks and the chunks are sequentially processed, left-to-right, where information from all chunks observed so far is consolidated into a summary. The response to a query (if there is one) is calculated based on the aggregate summary of the text. Despite the simplicity of the approach, the method performs well across various models and tasks.
Strengths
* Idea is quite simple and seems top be effective * Positive results on many tasks and benchmarks * It is interesting that even long context models capable of processing lengthy inputs benefit from the proposed method, which reinforces the hypothesis that sifting the relevant information from lengthy inputs directly can be difficult. Consistent improvements are observed on Claude models. * Appreciate the ablations including effect of direction (left-to-right, right-to-levet) and lost in the middle experiment. * Seems to work especially well when the inputs are long.
Weaknesses
* Technical depth is limited * Framing/presentation is somewhat misleading - From Figure 1, is summarization all that’s being done? Is CoA is just a fancy term to describe this sequential summarization process? - ‘Chain of Agents’ makes it sound like there are different agents doing different tasks, which is misleading * Fairly local view of performance results. Only comparisons against RAG/Vanilla baselines are presented. How about comparisons to best published numbers on these datasets? * Clarity issues - Figure 1 does not explain the architecture well - 127: W1 generates related evidence useful for answering question - Without knowing CU1, the reader cannot verify this. Basically unable to follow the discussion in 126-134 without knowing what CU1,2,3 are.
Questions
* How sensitive is the model to choice of the size of chunks? * Was the RAG baseline properly optimized? * Table 2 should also compare against RAG models. For some practical applications, RAG could be more beneficial due to inference speed.
Rating
5
Confidence
3
Soundness
3
Presentation
3
Contribution
3
Limitations
Limitations were discussed
Dear Reviewer, Since we are approaching the end of the discussion period, we are wondering if you have had the chance to review our response to your feedback. We would like to kindly inquire about the extent to which we have successfully addressed the concerns outlined in your review. We greatly value your feedback and would appreciate any further questions or comments you might have. Thank you for your time and consideration. Sincerely, All Authors
I appreciate the author's clear and point-by-point rebuttal, as well as the supplementary experiments provided. However, after thoroughly re-examining the paper, the rebuttal, the feedback from the other reviewer, and the related works in this area, I still have **significant concerns** regarding the novelty and the experimental validation of this work. First, as I mentioned in my initial review, the novelty of this submission remains my primary concern. I agree with reviewer *vGWD*'s comment that "*the technical depth of this work is limited.*" Although the authors compare LongAgent and RecurrentGPT in their general response, asserting that "LongAgent **does not allow communication between agents**, making it hard for each agent to understand what is happening with others," various communication mechanisms exist in multi-agent systems (see Section 3.3 in [1] and the vertical and horizontal multi-agent architecture definition in [2]). While CoA uses decentralized communication and LongAgent employs centralized communication, based on my understanding, the central agents in centralized communication essentially function as the communication units described in your paper. Additionally, you mention that LongAgent identifies the decomposition of the input question as a limitation. Could you clarify how CoA addresses this issue? From my understanding, CoA does not involve the decomposition of the input question. Furthermore, as the authors stated, "Regarding RecurrentGPT, although it uses a chain structure to perform long story generation, the task and motivation are different." However, this further diminishes the novelty of the proposed method. In my view, many components of the proposal are common practices in multi-agent system design and long-content generation/processing. The work appears to be a fine-tuned combination of existing methods rather than a deeply innovative approach. In my opinion, the limited novelty does not meet the standards of NeurIPS. Second, regarding the stronger baseline, the authors only tested LongAgent in the NIAH PLUS. Given that LongAgent is closely related to your work and was published three months before your submission, it should be comprehensively tested as a stronger baseline across all benchmarks you tested. Without this, it is difficult to claim that your communication unit or designed agent system is more efficient. Overall, I appreciate the effort the authors have put into the rebuttal. Unfortunately, it does not address my primary concern regarding the work's novelty. I remain open to further discussion with the authors in the coming days. References: [1] Guo, Taicheng, et al. "Large language model based multi-agents: A survey of progress and challenges." arXiv preprint arXiv:2402.01680 (2024). [2] Masterman, Tula, et al. "The landscape of emerging ai agent architectures for reasoning, planning, and tool calling: A survey." arXiv preprint arXiv:2404.11584 (2024).
I would like to thank the authors for the response. My concerns are adequately addressed. Thus I have improved my score to 5.
Decision
Accept (poster)