Cross-model Control: Improving Multiple Large Language Models in One-time Training

Summary

The development of the Cross-model Control method, which enables the use of fine-tuning outcomes from one model to improve other models through a portable tiny language model, effectively reducing training costs and computational resources. The introduction of a novel token mapping strategy called prefix match with minimal edit distance (PM-MinED) to adapt the tiny model to different vocabularies of target LLMs. Extensive experimental validation showing the effectiveness of CMC in common LLM optimization tasks like instruction tuning and unlearning, demonstrating substantial improvements over existing methods. This approach represents a significant advancement in the field of LLM optimization, offering a scalable and resource-efficient solution for enhancing multiple models simultaneously.

Strengths

The significance of this work lies in its potential impact on the practical deployment of LLMs. By enabling one-time, cost-effective optimization of multiple models, CMC addresses a critical bottleneck in the application of LLMs—resource intensity. This is particularly relevant in scenarios where computational resources are limited or where frequent re-training of models is not feasible. Additionally, the introduction of the PM-MinED strategy for token mapping broadens the applicability of the tiny delta model to diverse model architectures and vocabularies, thereby enhancing the utility of the proposed method in real-world applications.

Weaknesses

The effectiveness of the Cross-model Control hinges significantly on the observed similarity in logit shifts across different models post-fine-tuning. This assumption may not hold in situations where models diverge significantly in architecture or training data, potentially limiting the applicability of the method in more diverse or adversarial settings.

Questions

Generalizability Across Diverse Models: How well does the Cross-model Control (CMC) method generalize to models with architectures or training procedures significantly different from those used in your experiments? For instance, how would CMC perform with emerging architectures like Transformer-XL or models trained with novel objectives?

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors of the paper "Cross-model Control: Improving Multiple Large Language Models in One-time Training" have made an effort to address the limitations of their work, as well as discussing the broader societal impacts. However, there are areas where the discussion could be enhanced to provide a more comprehensive overview: Scalability and Efficiency: The computational requirements and potential bottlenecks of implementing the PM-MinED strategy, especially across models with large and diverse vocabularies, are not thoroughly explored. Bias and Fairness: There is a lack of discussion on how the method might influence or propagate biases within the trained models. Given the potential for large language models to perpetuate or amplify biases, this is a critical area that needs addressing. Robustness and Error Analysis: The paper does not provide statistical robustness measures such as confidence intervals or error bars, which are essential for validating the reliability of the results. Suggestions for Improvement Expand on Computational Challenges: Include a detailed analysis of the computational efficiency and scalability of the PM-MinED strategy. Discuss potential strategies to optimize this process for larger-scale applications.

Summary

Large language models face a series of common optimization requirements under specific applications or ethical standards. Existing methods only optimize one target model at a time, which requires changing model parameters or adding new parameters. Authors found that the logit changes of different models before and after fine-tuning are very similar. Based on this finding, the CMC method introduces a small language model with a very small number of parameters, and trains it together with a frozen template LLM to enable the small model to learn to change the output logit of LLMs. In order to make the delta model suitable for models with different vocabularies, the paper proposes a novel token mapping strategy called PM-MinED, which is a lightweight method for aligning the vocabulary of user LLM and delta model at the token level.

Strengths

1. The proposed method is novel and interesting. Unifying different training goals into one tiny model is interesting. 2. The proposed method has practical usage. It has significant potential to solve the problem of fintuning LLMs to fit different goals, especially for those users without enough resources. 3. The source code is given for reproducibility. 4. The experiment results show significant improvements of the proposed method.

Weaknesses

1. Authors should provide more details about the hyper-parameter settings. Do the hyper-parameter is chosen as a suitable value with respect to different methods? The delta model is training with longer epochs than LoRA, which may make the delta model learn more than LoRA. 2. Figure 2 is not clear to me. It seems that the logits are sorted. So, how does this figure illustrate the similarity? 3. The Figure 3 is somehow misleading. Are output logits both fed into the user LLM and the Delta Model, or the raw training inputs are fed into the user LLM and the Delta Model? 4. It is under-explored that whether the Delta model harms the original ability of user LLMs. 5. The hardware configuration required to train and inference with the delta model is not provided.

Questions

See weaknesses.

Rating

5

Confidence

3

Soundness

3

Presentation

2

Contribution

3

Limitations

See weaknesses.

Summary

This paper proposes a novel training method named CMC, designed to improve performance for multiple LLMs in one-time training, thereby reducing training costs by reusing fine-tuning outcomes. The core approach of the paper consists of three steps: 1. Introducing a portable tiny language model (delta model) with a small number of parameters, trained alongside the frozen template LLM, to enable the delta model to alter the output logits of the LLM. 2. Introducing a novel token mapping method called PM-MinED, which aligns the vocabularies of the delta model and the user LLM. 3. Sharing the delta model with other user LLMs to simulate the effect of fine-tuning. The author conducts extensive experiments on instruction tuning and unlearning tasks to demonstrate the effectiveness of the proposed method. Additionally, the paper highlights the significant potential of small language models in assisting large language models.

Strengths

1. This paper is well-motivated. The author identifies the pain points of existing model optimization methods that are not portable, and provides a detailed analysis of the fine-tuning similarity between different models, effectively leading to the core method. 2. CMC is simple yet effective. The delta model serves as a parameter module that can be ported to models with different parameter scales and vocabularies, still achieves impressive performance. Its performance significantly surpasses baseline methods that can only be ported within the model family, and it achieves similar performance to the non-portable method LoRA in the unlearning task. 3. CMC has strong generalization, demonstrating good performance when applying delta models of different parameter scales to user LLMs with different parameter scales and vocabularies. It also exhibits good robustness, being able to effectively mitigate overfitting and underfitting phenomena through the adjustment of the strength coefficient. 4. This paper showcases the great potential of small models in assisting super large models, which is quite exciting! 5. The author has open-sourced the complete implementation process, which is easy to reproduce.

Weaknesses

There are no apparent weaknesses, just some minor concerns. 1. In the vocabulary of language models, in addition to tokens that make up most of the words, there are still many special tokens. Mapping between special tokens from different vocabularies may require a significant amount of manual annotation. 2. The author needs to clarify how to represent a token that does not exist in the delta model's vocabulary when the user LLM generates it during inference. I'm concerned that if the token is directly encoded using the delta model's tokenizer, it may not adequately represent the information of that token. 3. The author would do well to include the pseudocode of the algorithm in the paper, which would clearly demonstrate the details of the inference part.

Questions

1. As mentioned in the weaknesses, during the inference stage, if the user LLM generates a token that does not exist in the delta model's vocabulary, what is the input to the delta model when predicting the next token? 2. In the preliminary section, the author indicates that shifts in logits across different models are similar. Can we use a delta model to directly fit the logits shifts of existing models, such as fitting the logits shifts of Llama2-7b-chat and Llama2-7b-base? Will token-level logits shifts provide more fine-grained supervision signals, leading to better performance?

Rating

8

Confidence

5

Soundness

4

Presentation

3

Contribution

4

Limitations

The authors have adequately addressed the limitations and risks of their work.

Summary

This paper investigates an interesting problem in fine-tuning LLMs: how to reuse the fine-tuning outcomes of one model for other models to reduce training costs. The authors provide an important empirical finding: different models exhibit highly similar logit shifts before and after the same fine-tuning process. Based on this, the authors propose Cross-model Control (CMC), a method that improves multiple LLMs in a single training session using a portable tiny language model. Specifically, the authors integrate a tiny language model with minimal parameters, training it alongside a fixed template LLM. This tiny model gains the ability to alter the output logits of the LLM, making it applicable to other models. Experiments demonstrates its effectiveness.

Strengths

1. How to improve multiple LLMs in one-time finetuning is really an interesting topic for me. 2. The method is well-motivated, reasonable, and experimentally validated. 3. The code has been provided, which is excellent.

Weaknesses

1. As the authors discuss in the limitations section, if the user’s LLM vocabulary includes symbols from languages not covered by the delta model’s vocabulary, the logits for these different language symbols will not be adjusted accordingly. While I don’t expect the authors to overcome this limitation, it does imply an underlying assumption in the proposed method. I would like the authors to explicitly state this assumption in the methods section. 2. Because the findings in Figure 2 are crucial for proposing the method, I would like to see not only the qualitative analysis provided in Figure 2 but also a quantitative analysis. This should include the similarity of logits shifts for different models fine-tuned on the same dataset, as well as the similarity of logits shifts for the same and different models fine-tuned on different datasets.

Questions

See "Weaknesses".

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors provide a discussion on limitations.

Summary

The paper introduces Cross-model Control (CMC), a novel method to improve multiple large language models (LLMs) in a single training session using a portable tiny language model. The core idea hinges on the observation that the logit shifts before and after fine-tuning are similar across different models. Leveraging this, CMC trains a tiny model alongside a frozen template LLM to adjust the logits output by LLMs. The paper also proposes a token mapping strategy, PM-MinED, to make the tiny model applicable to models with different vocabularies. Extensive experiments on instruction tuning and unlearning tasks demonstrate the effectiveness of CMC. The paper provides the code for their implementation.

Strengths

Originality: The paper introduces a novel approach to improving multiple LLMs simultaneously, which addresses a significant challenge in the field. The use of a portable tiny model to adjust logits across different models is innovative. Quality: The method is thoroughly tested on two important tasks—instruction tuning and unlearning. The experiments are well-designed and demonstrate the effectiveness of the approach. The analysis of logit shifts and the detailed comparison with baseline methods like LoRA and Proxy-tuning add to the robustness of the study. Clarity: The paper is well-written with clear explanations of the proposed method, experimental setup, and results. The figures and tables effectively illustrate the findings, making the concepts easier to grasp. Significance: The proposed method has substantial implications for the efficient fine-tuning of LLMs, especially for model owners with limited data and computational resources. The ability to apply fine-tuning outcomes across different models can lead to significant cost reductions and broader applicability of advanced LLMs.

Weaknesses

Token Mapping Strategy: While PM-MinED is introduced to handle different vocabularies, its effectiveness might be limited when dealing with highly diverse or specialized vocabularies not covered in the training data. This could impact the generalizability of the method. Performance on Larger Models: The performance of the tiny model on extremely large LLMs (70B llama 2 for example) can sometimes lead to overfitting, as indicated in Section 4.3. The paper acknowledges this but does not provide a concrete solution beyond adjusting the model size and strength coefficient. Limited Error Analysis: The paper lacks a detailed error analysis, particularly concerning the cases where the method fails or underperforms compared to other approaches like LoRA. A deeper investigation into these scenarios could provide valuable insights for future improvements.

Questions

Token Mapping Generalizability: Can you provide more details on how PM-MinED performs with highly specialized or domain-specific vocabularies? Have you tested the method on such datasets? Handling Overfitting: Beyond adjusting the model size and strength coefficient, are there other strategies you considered to mitigate the overfitting of the tiny model when applied to larger LLMs? Error Analysis: Can you provide more detailed error analysis or failure cases where CMC underperforms compared to other methods? This would help in understanding the limitations and potential areas for improvement.

Rating

6

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors have addressed the limitations of their work adequately, highlighting the constraints related to the scope of the tiny model's vocabulary and its impact on performance. They also discuss the potential for overfitting when applying the tiny model to very large LLMs and the need to adjust model parameters carefully.

Thanks for responses

Thanks for efforts in responses. After reading the response, I'd like to raise my score as 5.

Thanks for you response

I have thoroughly reviewed all the comments and the author’s responses, and I will remain postive of this submission.