Summary
The paper proposes MacNet, a multi-agent collaboration network. Inspired by neural scaling laws, the authors investigate how the number of agents and the communication topology between agents impacts performance when agents are made to collaborate on a given problem. Specifically:
- MacNet creates a Directed Acyclic Graph (DAG) of agents that attempt to solve a given task in analogy to a neural network. The **nodes and edges are agents**, where the nodes are "executors", which accomplish tasks, while the edges are "instructors", which evaluate the task and provide feedback. Agents are allowed to **interactively and iteratively refine** their subtask solutions, where an instructor asks an executor to refine their proposed subtask solution. To avoid **content explosion**, nodes in the network are expected to only pass on the proposed solution, not its entire reasoning, and they are also expected to aggregate incoming solutions. Importantly, MacNet does not have a specific topology; the method can be applied to many different types of topologies, including random topologies.
- To evaluate the performance of MacNet, the authors evaluate on MMLU, HumanEval, SRDD and CommonGen-Hard. They compare MacNet variants based on different DAG topologies against Chain of Thought, AutoGPT, GPTSwarm and AgentVerse. Their method exhibits significantly better performance on MMLU, mostly worse performance on HumanEval, slightly better on SRDD and sometimes better performance on CommonGen.
- They compare the performance of different topologies against each other, finding that random topologies often perform statistically better.
- They observe the performance of different topologies across multiple network sizes, demonstrating a non-decreasing performance trend.
- They perform an error analysis on one of the topologies at different sizes, finding longer tails when the number of agents increases which diversifies the solutions, making the proposed solutions more comprehensive.
Ultimately, they conclude that agent collaboration could bypass the need for retraining by enabling inference time performance scaling through scaling multi-agent collaboration.
Strengths
# originality
* The authors study the scaling of the number of agents during inference, a relatively **novel, unexplored problem**.
* The authors **extend the interaction DAG setting** to a functionally bipartite structure with agents as nodes and edges, and **present representative topologies** that capture prevalent DAG structures
* The authors **address the limitations of prior work** by avoiding content explosion
# quality
* The authors describe the problem setting with the appropriate technical language, as well as **motivating theoretical justifications** for MacNet's advantages over existing solutions.
* The authors compare against a set of **timely and relevant baselines** on ** diverse datasets**.
* The authors use **identical and reproducible settings** across methods.
* The authors provide **multiple perspectives** on their results, connecting observed phenomena to novel and documented insights.
# clarity
* The paper is **well-structured**, making it easy to read and follow along.
* The paper uses **interesting visualizations** to communicate interesting insights, such as longer tails with more agents.
* The paper employs **precise formal language and notation** to effectively communicate their points.
# significance
* The paper has a significant impact on **LLM-based applications**
* The paper is very **timely** as the research on multi-agent systems begins to regain the interest of the machine learning community.
* The paper offers **interesting insights** into collaborative inference-time structures across many agents, which is relevant to ongoing discussions around agent proliferation.
* The paper outlines **future directions of work**.
Weaknesses
# Problem Setting
* **collaborative emergence is undefined**, even though it seems to be a central concept of the paper.
# Results
## Major
- **Lack of transparent methods**: The use of the closed source line of GPT models hinders the analysis. The authors take the stance that the multi-agent scaling laws are analogous to the neural scaling laws, but we do not know much about the underlying models. It would be more judicious to compare open-source models, for which we know the FLOPs and parameter counts. This could allow for comparisons such as a single 7B model vs a MacNet of 7 1B models. An additional point here is that we do not know what type of post-training the GPT models undergo, which could help in the collaboration. The main point of this criticism is the following: **the use of a closed-source method in your analysis effectively hinders the potential for your results to generalize to other LLM-based agents**. We don't know the impact of model size, architecture or training strategy for the GPT models, so we can't know how the established analysis and scaling laws transfer to other architectures.
- **Entangled sources of improved performance**: Since the actual FLOPs used are not kept track of, it becomes impossible to tell whether the improved results are due to additional computational resources spent on figuring out a solution. It would be desirable to **isolate the impact of the multi-agent collaboration** from other factors, such as how much compute is poured into a given solution. Otherwise, it could be that a given MacNet works better because it pours more resources and time into a solution. While this result would be interesting in itself, it would shift the significance of the approach mostly onto methods for scaling inference-time compute, at which point you'd have to normalize for how much time and compute is spent on a given solution. You would then also have to compare against other non-multi-agent methods for scaling inference time compute. Instead, the diversity and comprehensiveness of the solutions output by a MacNet could be the subject of a greater analysis, while being more direct about the resource requirements.
## Minor
- **Lack of scaling law analysis**: The scaling laws, although the sigmoid functional form is described, are not actually fit, nor analyzed. Furthermore, there is little to no discussion as to *why* the scaling laws should take a sigmoidal form in this case.
# Overall
The presentation of the paper is solid. The writing is easy to understand and the use of technical language is judiciously sparse. The problem setting is good and the contribution would be too, but the soundness of the claims is affected by the limits to the results outlined above. This unfortunately impacts the value of the contribution itself, as the results are not wide-ranging.
Questions
- Doesn't the interative feedback between instructors and executors make the graph not directed? You could get stuck in a loop there.
- Have you looked into letting the agents attempt to determine what a good topology is? I.e. Agents specify to which other agent they believe they should send their result.
- If quality is the average across all tasks, does each benchmark have the same amount of tasks? Or did you average specifically task-wise, or across all 4 datasets?