Summary
This paper introduces an approach called DyLAN that incorporates a two-stage paradigm: i) Team Optimization, the temporal feed-forward neural network (T-FFN) that selects most contributary agents for the given task and ii) Task solving, the T-FFN facilitate agents to communicate with each others to solve a bunch of a diverse of tasks. To further enhance the efficiency, DyLAN uses early stopping mechanism to terminate the conversation when most of agents in a single layer agree to have a consistent answer. In the experiment section, the DyLAN is proved to outperform with baselines such as LLM-Blender, LLM Devate, ReAct, and Reflexation, etc, on code generation, decision making, and general reasoning, etc, and shows that DyLAN.
Reasons to accept
1. Solid Experiments: The paper provides a robust experimental methodology, demonstrating DyLAN's effectiveness across a range of downstream tasks such as coding, general knowledge quetsion answering, math etc. It shows that cost DyLan can enhance the performance of the above mentioned tasks while keep low cost on computation with respect to reasonable times of function call.
2. The idea of DyLAN is novel, it trains a dynamic temporal feedforward network to select the imporatant agents, and the early stopping seems to be effective to reduce the cost on long-term conversation.
3. The ablation study is thorouh.
Reasons to reject
1. **Extended Query on the Selection and Performance of Agents in the Initial Pool**: "Upon close examination of Figure 5 within the manuscript, there appears to be a lack of clarity regarding the roles adopted by the other agents involved in the code generation task. It is noted that the 'programmer' and 'algorithm developer' agents seem to outperform the rest, which might suggest that these two agent roles are disproportionately influential in the task's outcome. This observation raises the question of whether these two agent roles alone suffice for the task at hand. Similarly, in the mathematical reasoning tasks, the 'programmer' and 'mathematician' agents are indicated as the primary contributors, prompting the inquiry of whether including agents with roles such as 'doctor' or 'economist' might be an overcomplication in the context of mathematical tasks. If the initial agent pool is curated for specificity to different tasks, then the inclusion of seemingly irrelevant agents could be considered redundant and may detract from the efficiency of the overall system. Could the authors provide insight into the rationale behind the selection of these varied roles and how their inclusion impacts the performance and generalizability of the DyLAN framework?"
2. Upon reviewing Figures 5 and 6, it is observed that layers with an even number typically assume a critical function, evaluating the responses generated by the preceding odd-numbered layers. This pattern suggests that the effectiveness of the DyLAN framework may be contingent upon the utilization of highly curated prompts tailored to fit each agent's distinct roles and scenarios. The observed phenomenon raises concerns regarding the scalability and flexibility of the system, particularly in scenarios where the diversity of tasks and roles is substantial. It would be beneficial for the authors to address the necessity for such detailed prompt curation and discuss potential strategies to streamline this process, ensuring that the framework remains adaptable across a wide array of tasks without compromising performance.
However, I find the paper to be of high quality overall; if the authors can provide clear and satisfactory responses to the queries raised, I would be willing to increase the points.