ReFT: Representation Finetuning for Language Models

Summary

The paper introduces Representation Finetuning (ReFT) - a family of methods to learn interventions directly on model representations, rather than model weights. The authors compare ReFT to Parameter Efficient Finetuning (PEFT), and find that it yields similar performance while being significantly more parameter-efficient.

Strengths

- The paper is well-written. - The main ideas in the paper are clear and easy to understand. - Novel and impactful contribution. - The paper presents a new paradigm of learning modifications to representations, rather than learning modifications to weights. This new paradigm is more parameter-efficient, while appearing to have a similar level of expressivity. PEFT methods have been incredibly impactful, and I could see ReFT being similarly impactful.

Weaknesses

- Lack of representation-editing-based baselines - The paper focuses on comparing ReFT to PEFT methods. Appendix B discusses existing representation-editing methods, and casts them in terms of the ReFT framework. - I think it would strengthen the paper to compare ReFT and DiReFT to these existing representation-editing methods. I am curious to understand the difference in performance of an intervention learned by gradient descent (e.g. ReFT) vs an intervention learned by contrastive pairs (e.g. activation addition, RepE). - Some unclear presentation - Line 102 - What does "the hidden representation created at row $i$ and column $k$" refer to? As far as I can tell, this does not integrate with the previous notation defined in the second paragraph of Section 3. I assume this refers to layers and positions - if this is the case, then it would be clearer to say so explicitly. - Inconsistent variable names - I think the presentation would be clearer if variable names were used across sections. - Section 3.1 uses $\mathbf{b}$ to represent a hidden state, whereas Section 3.2 uses $\mathbf{b}$ to refer to a bias vector. I think this can be easily fixed by using $\mathbf{h}$ to represent the hidden state in Section 3.1. - Section 3 uses $m$ to represent the number of layers, whereas Section 3.2 uses $m$ to refer to the length of the output sequence $\mathbf{y}$. - Typo in Table 17 (?) - LLaMA-7B/DiReFT/AQuA: 221.3. - It might be worth double checking your numbers more generally if tables are not generated by code. - Other suggestions - Section 3.2 - The paper could benefit from a motivation of the LoReFT expression, and how it was selected over the other expressions mentioned in Appendix E. - Define dropout more explicitly - What does dropout refer to in this case? My assumption is that with probability $p$, the intervention is not performed. But in the cases where the intervention is not performed ("dropped out"), what is being optimized? - Include baseline generations in Appendix I - Only LoReFT outputs are given in Appendix I - it's hard to interpret these generations without having baselines to compare them to. For example, even examining the difference between the baseline (no intervention) and LoReFT would be helpful, particularly for the long-form generations.

Questions

- Practical recommendations for using ReFT: - Appendix E mentions that there is not one clear best expression of ReFT. Which variation would the authors recommend individuals use, and why? - Is there a recommended methodology for determining hyperparameters in practice? - In what scenarios should one use PEFT vs ReFT vs other inference-time interventions?

Rating

7

Confidence

3

Soundness

3

Presentation

3

Contribution

4

Limitations

- The authors address the following limitations: - The classes of models evaluated is limited. - Hyperparameter selection seems fairly complicated, and automating this selection will be valuable for future adoption of the methodology.

Summary

This work proposes a novel method for fine-tuning language models (LM) called Representation Fine-tuning (ReFT), which updates only a small number of parameters. Unlike existing parameter-efficient fine-tuning methods such as LoRA, ReFT enables fine-tuning with minimal parameter updates by learning small interventions in the representation of specific layers and token positions. The authors introduce Low-rank Linear Subspace ReFT as a method within ReFT, demonstrating experimentally that it can fine-tune LMs effectively with fewer parameter updates compared to existing parameter-efficient fine-tuning methods.

Strengths

- **Originality**: While representation engineering has been utilized in various works, applying it to parameter-efficient fine-tuning is unprecedented. The motivation from interpretability works is also interesting. From the perspective of originality, this work is commendable. - **Quality**: The proposed method’s effectiveness is empirically validated on multiple LMs based on Llama and RoBERTa across approximately 20 datasets and four tasks, proving its practical applicability. Additionally, the authors provide extensive experimental results with various hyperparameters in the appendix, offering sufficient reference data for future users and making it easy to identify scenarios requiring caution. Thus, this paper can be considered a complete piece of work. - **Clarity**: The paper is written very clearly, including appropriate figures to facilitate easy implementation by the reader. - **Significance**: This paper holds significant value as it suggests a more effective way to fine-tune LMs through interventions in representation rather than model weights, unlike existing PEFT methods. It can serve as a drop-in replacement for the widely used LoRA, potentially having a substantial impact on future LM fine-tuning research.

Weaknesses

- **Quality**: Although the extensive experimental results in numerous settings sufficiently demonstrate the method's significance compared to existing methods, including results from models other than Llama, such as Mistral or Phi, would emphasize the method's applicability in various scenarios. - **Significance**: The need for extensive hyperparameter optimization to decide which layer and position of hidden representation to apply the intervention function is a potential weakness. This issue is well explained and mentioned in the Limitations section.

Questions

### Questions Practically speaking, when do the authors believe this method should be attempted instead of LoRA? In other words, in what situations is the LoREFT most appropriate? Based on the experimental results, it appears that using the LoREFT might be unsuitable for achieving high performance on inference tasks such as GSM8K.

Rating

9

Confidence

4

Soundness

4

Presentation

4

Contribution

4

Limitations

The authors adequately addressed the limitations in Section 5.

Summary

The authors propose an alternative PEFT method based on representation intervention techniques that are used in interpretability research. They evaluate their method in a variety of settings including multiple architectures and finetuning dataset families.

Strengths

* The method presented by the paper uses an order of magnitude fewer parameters than comparable baselines (e.g., LoRA, DoRA) while still maintaining comparable accuracies after finetuning. * The method achieves consistently stronger performance than the baselines on commonsense reasoning tasks * Evaluation was thorough: the authors evaluated multiple architectures across scale and multiple dataset families.

Weaknesses

LoRA and other baselines can adjust the number of tunable parameters via the rank parameter. The authors should evaluate specifically how their method compares to the baselines when there are a comparable number of tunable parameters, e.g, by lowering the LoRA rank. The original LoRA paper suggests that performance can sometimes increase (and often, at least, not decrease) when lowering the rank. Given that the ReFT intervention occurs on a fixed set of positions, the authors should evaluate if ReFT is effective even in long-context settings. While there are performance improvements on the commonsense reasoning tasks, performance decreases on other tasks (sometimes fairly substantially, e.g. in Table 2), which limits the applicability of this method. While the authors evaluate multiple settings, it would be interesting to present results on a wider class of evaluations and finetuning datasets to evaluate where ReFT would be preferable to other datasets. (Note: While this would be interesting and would make for a stronger paper, I do not believe it is necessary for a solid paper).

Questions

What is the memory footprint of ReFT in comparison with other methods? Since the intervention occurs on a fixed number of prefix tokens, what happens when the prompt prefix is shorter than the number of intervened positions?

Rating

6

Confidence

4

Soundness

4

Presentation

4

Contribution

3

Limitations

The authors adequately address the limitations.

Summary

This paper proposes representation finetuning for efficient tuning or intervening for task-specific representations in models while keeping the base model frozen. They define LoReFT and unify several current representation intervention methods under their framework. They conduct extensive experiments on several types of NLP benchmarks and models to demonstrate the efficiency and better understand the effectiveness of ReFT.

Strengths

1. The paper proposes a new and potentially useful paradigm for efficient fine-tuning of model representations for specific tasks. Their proposed LoReFT inherits the merits of previous representation intervention method and operates on a low-rank subspace to control the the representations. They also put LoReFT under a bigger framework of representation intervention and discuss its relationship with other previous method. They provide good insight on understanding the development process of current representation-level interpretability work. 2. Their experiments are extensive and solid. They successfully demonstrated the efficiency of their ReFT, as well as the effectiveness to some extent. I admire that the authors are willing to show the limitation of performance of their method on some types of tasks to faithfully argue the benefits of their method. 3. They have an open-sourced package for reproducing the whole pipeline. They also have detailed documentation of their hyperparameter tuning process. I think this is especially important for new methods like ReFT which I imagine would requires some deep understanding to tune the hyperparameters. 4. They have some interesting intervention examples in the Appendix. They also provide very good ablation of LoReFT design in the Appendix.

Weaknesses

1. I feel the hyperparameter tuning is still pretty heavy althoughtthe authors try to show that they only need to finetune the model on one task from a specific type and use the set of hyperparamters for other tasks in the same type. I expect to see more results on how robust the hyerparameters are across models in the wild. For example, if we find a set of hyperparameters for Llama 7B, will that generalize to Llama2 7B or models of the same size? I also want to see how much the variance is for the best set of hyperparameters for different models and tasks. 2. Sometimes I expect to see how ReFT would fit the current pipeline of SFT + RLHF alignment paradigm. Some results, for example, on the instruction tuning datasets, are lag behind the current SOTA by too far. I'm not saying the method should achieve SOTA but would expect to see the possibility of it to push the new boundaries in the current context of LLM fine-tuning.

Questions

1. What do the prefix and suffix refer here for classification tasks and generation tasks in your hyperparameter tuning descriptions? 2. You might not have experiments on that but just out of curious if ReFT can be used to optimize the objective in preference-based learning like the Bradley-Terry model in DPO and how it would perform. 3. For the ultrafeedback fine-tuning, do you select the best responses to do MLE? As this is usually used as a preference learning dataset.

Rating

6

Confidence

4

Soundness

3

Presentation

4

Contribution

3

Limitations

yes

Reply to the rebuttal

I appreciate the clarifications the authors made around my questions. I like the one that mentioning quicker iteration for data selection on alignment pipeline using ReFT. I would keep my overall positive feedback on the paper and will keep my score unchanged.