Agent Skill Acquisition for Large Language Models via CycleQD

Responses to Reviewer kupr (Part 1)

We thank the reviewer for taking the time to provide thoughtful comments and suggestions. We are happy to see that the reviewer agrees our work delivers strong empirical results and provides studies validating the design choices. The following are our responses to the comments and questions, please see the details in the revised PDF. **About the theoretical foundation of elite sampling and hyperparameter selection.** We have added text in Sec 3.1 to provide a foundation for elite sampling, and have clarified how the hyperparameters were selected in Sec A.1.3. In summary, previous research in QD explores sampling methods that expand the frontier not only in Q but also within BCs, allowing for more diverse behavior exploration and higher quality solutions. For example, the paper “Multi-objective Optimization-based Selection for Quality-Diversity by Non-dominated Sorting” evaluates whether each elite is on the Pareto frontier by checking if it is dominated within each behavior space and samples from elites on the frontier. Inspired by this work, we also incorporate behavioral space to sample elites, with two main differences from prior approaches. (1) Both Q and BCs are task performances and are treated equally in a cyclic process. (2) To achieve higher performance, pushing the frontier in a high-performance direction rather than a low-performance one is ideal. Therefore, we aggregate Q and BC values into a single metric to drive the frontier towards higher performance. Our hyperparameters are determined in an arbitrary manner: we manually picked a set of values, ran CycleQD for a small number of generations with each setting and picked the best. We believe more sophisticated methods such as grid search will generate better results. **About computation costs.** We have added a comparison between CycleQD and gradient-based fine-tuning in A.1.7. As a summary, the gradient-based fine-tuning took ~200 GPU hours, and CycleQD took ~410 GPU hours (excluding the experts training time). We wish to point out that (1) the extra time spent on CycleQD is mostly due to agentic task evaluations rather than the optimization, (2) due to GPU memory constraints, the merging process was carried out on CPU, (3) CycleQD produces an archive of models whereas FT generates only one. Furthermore, it is important to note that fine-tuning is a mature and well-established approach with software implementations that are optimized for efficiency. By contrast, the proposed method was developed primarily to assess its performance, and as such, there is ample room for further efficiency improvements.

Responses to Reviewer kupr (Part 2)

**About comparison with NSGA-II.** We have conducted a comparison with NSGA-II and have presented the setups and results in Sec 4.1. In summary, as is shown in the table below, while NSGA-II outperforms other model-merge-based baselines, it remains inferior to CycleQD. Additionally, it is worth noting that NSGA-II is known to have two limitations: (1) performance degradation as the number of objectives increases, and (2) reduced effectiveness of the crowding distance mechanism in high-dimensional spaces. CycleQD does not suffer from these limitations, further emphasizing the generality and robustness of our method. | # | Methods | MBPP | DB | OS | Avg | |----|-------------------------------|------|------|------|------| | 7 | Merging (w/o learning) | 72.9 | 24.7 | **42.6** | 46.7 | | 8 | Merging (learning w/ GD) | 69.3 | 41.2 | 29.6 | 46.7 | | 9 | Merging (learning w/ CMA-ES) | 69.3 | 41.2 | 30.2 | 46.9 | | 10 | Merging (learning w/ NSGA-II) | 75.9 | **42.4** | 36.4 | 51.6 | | 11 | CycleQD (Ours) | **76.4** | 38.2 | **42.6** | **52.4** | **About why cyclic optimization is more effective.** We share our insights as to why cyclic optimization is more effective in the following. For simultaneous optimization where aggregation of the objective functions are needed: as stated in the introduction, the tuning of task ratios in the objective function is often required, which can be challenging and highly problem-specific. Similar to coordinate descent, CycleQD effectively avoids this problem by using task metrics as both Q and BCs and cyclically optimize through each of them, ensuring balanced improvements without the need for explicitly weighted aggregation. For simultaneous optimization where objective aggregation is not necessary (e.g., NSGA-II): CycleQD’s cyclic optimization mechanism incorporates elements resembling non-dominated sorting. However, unlike NSGA-II, which handles all tasks simultaneously and relies on crowding distance for diversity, CycleQD alternates between tasks and naturally encourages diversity via the QD mechanism. These settings reduce interference between objectives and allows for deeper refinement of task performance. **About the theoretical basis for SVD mutation.** We have added texts at the end of Sec 3.3 to explain why SVD-based mutation. In summary, our SVD-based mutation provides two key advantages: (1) SVD focuses mutations on the fundamental building blocks of the task vector, which represent the essential components of agent skills. By constraining perturbations to these meaningful directions, we avoid the excessive exploration inherent in random perturbations, reducing the risk of overfitting. (2) The ability to manipulate individual components (i.e., $\omega$) offers finer control over $\theta_\texttt{child}$, enabling targeted modifications at a granular level. **About combining skills from vastly different domains and possibility to outperform SFT.** In addition to the three CS tasks, we have incorporated a VQA task into CycleQD in this revision (see A.1.5). As a vision-language modeling task, VQA differs significantly from computer science tasks, making this incorporation particularly challenging. During the CycleQD process, we extract the LLAMA3-8B-INSTRUCT component from LLAMA3-LLAVA-NEXT-8B and fuse it with other expert models. The results indicate that CycleQD achieves better VQA performance compared to its expert model, while maintaining higher performance on MBPP and OS than their respective experts. Although its DB performance is lower than the DB expert, CycleQD achieves the highest average performance overall.

Follow-up to Rebuttal

We would like to thank reviewer kupr once again for your time and efforts in reviewing our work. With our response and rebuttal revision, we made our best efforts to try and address the points raised in your constructive feedback. As the discussion period is coming to a close, we would really appreciate it if you could let us know if you found our response satisfactory and your thoughts on the revised manuscript. Please, do not hesitate to raise any further questions or concerns about our work.

Responses to Reviewer 8w66 (Part 1)

We are grateful for the reviewers’ detailed feedback and suggestions. We are also delighted to see that the reviewer thinks our method to be a novel solution to a critical problem. The following are our responses to the comments and questions, please see the details in the revised PDF. **About the presentation of Algorithm 1.** We have added text to refer to Figure 4 (now Figure 6 after revision) while explaining Algorithm 1. In addition, we have also added a background section and Figure 2 to introduce evolutionary algorithms, specifically MAP-Elites, before describing our method to get the readers familiar with the operations mentioned in Algorithm 1. We hope this addresses the reviewer’s concern. **About the meaning of $\gamma$ and the intuition of elite sampling.** The meaning of $\gamma$ and its related components are detailed on lines 189 and 194, with the reasoning behind the design elaborated in the same paragraph. We have also added more text to provide a theoretical foundation for elite sampling. We hope this addresses the reviewer's concern. **About the sign of $\omega$ in the crossover operation.** The model mixing coefficients $\omega$ do not necessarily have to be positive numbers. Task vectors play an important role in model performance on certain tasks. By allowing negative component weights, the merged model has more freedom in optimizing its task vectors. And this optimization is conducted via evolutionary search. We have added this explanation in Sec 3.2 to clarify this. **About the explanation of SVD-based mutation and its finer control.** We have added texts at the end of Sec 3.3 to explain why SVD addresses this problem, and also clarify how SVD-based mutation allows finer control. In summary, our SVD-based mutation provides two key advantages: (1) SVD focuses mutations on the fundamental building blocks of the task vector, which represent the essential components of agent skills. By constraining perturbations to these meaningful directions, we avoid the excessive exploration inherent in random perturbations, reducing the risk of overfitting. (2) The ability to manipulate individual components (i.e., $\omega$) offers finer control over $\theta_\texttt{child}$, enabling targeted modifications at a granular level. **About model aggregation design.** As the reviewer pointed out, the purpose of this paper is to create a single model that shows consistently high performance across multiple tasks minimizing performance degradation between tasks. CycleQD generated numerous elite models specialized for each task. While these models have the potential to be applied to various use cases, we aggregated them to obtain a “generalist” model. To this end, a model aggregation method is needed. While there are numerous ways one can choose for model aggregation. We adopted a simple convex combination, which created the weights via softmax of the task performances. This is an arbitrary choice, and other methods (e.g., simple average of the performances) yielded an aggregated model of similar performance.

Responses to Reviewer 8w66 (Part 2)

**About the value of experiments on SAM.** The purpose of this experiment is to demonstrate the generality of our method across different domains, which we believe is an important aspect of the paper’s contribution. While the primary focus of our work is on merging LLMs to acquire agentic skills, one of the key strengths of CycleQD lies in its domain-agnostic design (i.e., task metric is the only thing that is required and it serves as both Q and BC). By applying CycleQD to SAM, we aim to prove this point and highlight that the proposed method is not limited to language models but can be generalized to other foundation models in the future for more diverse agentic skills. **About adding more tasks to support the generality of CycleQD.** In addition to the three CS tasks, we have incorporated a VQA task into CycleQD in this revision (see A.1.5). As a vision-language modeling task, VQA differs significantly from computer science tasks, making this incorporation particularly challenging. During the CycleQD process, we extract the LLAMA3-8B-INSTRUCT component from LLAMA3-LLAVA-NEXT-8B and fuse it with other expert models. The results indicate that CycleQD achieves better VQA performance compared to its expert model, while maintaining higher performance on MBPP and OS than their respective experts. Although its DB performance is lower than the DB expert, CycleQD achieves the highest average performance overall. **About combining CycleQD with SFT.** We have further fine-tuned the model optimized by CycleQD with the entire datasets for Coding, DB, and OS (model 11 in the following table). However, the fine-tuned model exhibits lower performance compared to the original CycleQD model (model 10). This suggests that CycleQD enhances the model's capability to its limit. Additionally, the fine-tuned model's performance is lower than that of other expert models (models 3, 4, 5, 6), indicating that it may have fallen into overfitting. | Method | MBPP | DB | OS | Avg | |--------------------------------|------|------|------|------| | 3. Fine-tuning (Coding expert) | 70.4 | 21.2 | 20.7 | 37.4 | | 4. Fine-tuning (DB expert) | 65.8 | 42.4 | 28.5 | 45.6 | | 5. Fine-tuning (OS expert) | 66.3 | 0 | 30.4 | 32.2 | | 6. Fine-tuning (All) | 67.3 | 37.1 | 36.7 | 47.0 | | 10. CycleQD (Ours) | 76.4 | 38.2 | 42.6 | 52.4 | | 11. CycleQD + ALL | 68.8 | 30 | 26.3 | 41.7 | **About computation comparison between CycleQD and gradient-based fine-tuning.** We have added this comparison in A.1.7. As a summary, the gradient-based fine-tuning took ~200 GPU hours, and CycleQD took ~410 GPU hours (excluding the experts training time). We wish to point out that (1) the extra time spent on CycleQD is mostly due to agentic task evaluations rather than the optimization, (2) due to GPU memory constraints, the merging process was carried out on CPU, (3) CycleQD produces an archive of models whereas fine-tuning generates only one. Furthermore, it is important to note that fine-tuning is a mature and well-established approach with software implementations that are optimized for efficiency. By contrast, the proposed method was developed primarily to assess its performance, and as such, there is ample room for further efficiency improvements. **About sensitivity to the number of generations.** We have added a sensitivity study to reveal this. Specifically, we tested CycleQD performance every 100 generations and added the results in A.1.3. In Figure 5, the performance gradually increases over the generations. We don’t observe large oscillations in performance w.r.t the number of generations.

Responses to Reviewer mjB3

We appreciate the reviewers’ thoughtful comments and valuable insights. And we are grateful that the reviewer thinks our work to be very original. The following are our responses to the comments and questions, please see the details in the revised PDF. **About introducing background knowledge.** We have created a “Preliminaries” section (see Sec 2) to introduce the concepts in evolutionary algorithms and have added a figure to illustrate the flow. We specifically introduced concepts like crossover and mutation in the context of MAP-Elites to help paint the picture before diving into the details in Sec 3. **About the motivation for evolutionary approaches.** Our motivation is (1) to reduce the intricate design decisions and hyperparameter tuning, and (2) as an exploratory work, to show that evolutionary algorithms can be incorporated into LLM fine-tuning pipeline, which is currently dominated by gradient-based methods, as a compelling modification. We have added text in the introduction to emphasize (1), and (2) is stated on lines 94-95. We hope the added text would better clarify our motivation. **About formatting.** We reviewed [the ICLR 2025 submission template](https://www.overleaf.com/latex/templates/template-for-iclr-2025-conference-submission/gqzkdyycxtvt) and found no restrictions on using half-column figures. In addition, some accepted papers of ICLR 2024 use a half-column figure format (papers [1](https://openreview.net/attachment?id=eUgS9Ig8JG&name=pdf), [2](https://openreview.net/attachment?id=kmn0BhQk7p&name=pdf), [3](https://openreview.net/attachment?id=csukJcpYDe&name=pdf)). Based on this, we conclude that half-column figures are permissible. **About the number of tasks.** To show the general applicability and versatility of CycleQD, we have added a VQA task in addition to the three CS tasks (see A.1.5). One of the motivations for this work is to show that evolutionary strategies can be an effective component in the LLM post-training pipeline, and we believe that CycleQD has laid a foundation for future exploration with more tasks.

Responses to Reviewer tJqK

We thank the reviewer for the constructive feed and thinking of CycleQD as an innovative and scalable solution to merge agent skills. The following are our responses to the comments and questions, please see the details in the revised PDF. **About CycleQD’s applicability to other fields** In addition to the three CS tasks, we have incorporated a VQA task into CycleQD (see A.1.5). As a vision-language modeling task, VQA differs significantly from computer science tasks, making this incorporation particularly challenging. During the CycleQD process, we extract the LLAMA3-8B-INSTRUCT component from LLAMA3-LLAVA-NEXT-8B and fuse it with other expert models. The results indicate that CycleQD achieves better VQA performance compared to its expert model, while maintaining higher performance on MBPP and OS than their respective experts. Although its DB performance is lower than the DB expert, CycleQD achieves the highest average performance overall. **About a clearer illustration of the methodology** We have created a “Preliminaries” section (see Sec 2) to introduce the concepts in evolutionary algorithms and have added a figure to illustrate the flow. We also refer to this figure when introducing our CycleQD. We hope this describes our method clearer. **About ablation studies.** We have revised Sec 4.1.5 to better articulate the significance of our ablation studies and to highlight the following aspects. The cumulative improvement from baseline QD to our final CycleQD is substantial at 4.8 percentage points (from 47.6% to 52.4%). This is a meaningful improvement in the context of LLMs performing complex agent tasks, particularly considering our model approaches GPT-3.5-TURBO's performance despite having significantly fewer parameters. Each component shows clear contributions through carefully controlled experiments: - Introduction of CycleQD: +1.8% improvement over standard QD - SVD-based mutation: +2.3% improvement over no mutation - Elite sampling: +0.7% improvement over random sampling Most notably, our ablation studies reveal important qualitative insights about design choices. For instance, the comparison between mutation strategies (trials 1-3 in Table 2) demonstrates that naïve Gaussian mutation actually hurts performance (48.5% vs 49.4% with no mutation), while our SVD-based mutation shows clear benefits (51.7%). This validates the importance of our design choices and shows that improvements aren't merely additive - poor design choices can harm performance while our well-designed components work synergistically. **About real-world applications across different scenarios.** We have added a VQA task to demonstrate the general applicability of our method beyond the original CS tasks, highlighting the versatility of our approach across different scenarios. At the same time, we acknowledge the challenges of deploying an 8B LLM in real-world scenarios. While smaller models are computationally more feasible for experimentation, their capabilities are often limited compared to larger LLMs. Nonetheless, our work demonstrates that evolutionary strategies can be an effective component in the LLM post-training pipeline, laying a foundation for future exploration with more capable models.