Personalized Steering of Large Language Models: Versatile Steering Vectors Through Bi-directional Preference Optimization

Summary

The paper studied the problem of steering large language models through steering vectors on the neural network activations. The motivation is to find an effective steering method than state-of-the-art approaches (e.g., contrastive activation addition) and preserves the utility of the original LLM. To achieve this, the authors proposed a novel optimization objective to produce steering vectors through bidirectional preference optimization. This approach is directly applied to activations, and aims to enhance the likelihood of generating responses corresponding to the target behavior while simultaneously reducing the probability of responses associated with the opposite behavior. Empirical results show that the proposed method outperforms state-of-the-art LLM steering methods, and fine-tuning method (DPO), across multiple models. Ablation studies further shows detailed results on the steered model's utility, transferability, synergy of vectors etc.

Strengths

1. The paper studied an important and interesting problem of LLM steering 2. The proposed steering objective is novel and concise. The idea draws inspirations from preference optimization techniques from fine-tuning to the more light-weight approach of optimizing steering vectors in the activation space, which is very interesting and seems to show positive improvement empirically. 3. The paper presents a comprehensive empirical study of the performance of the proposed approach against state-of-the-art methods as well as more detailed ablation studies. 4. The paper is overall well presented and easy to follow.

Weaknesses

1. This could be a general challenge which also applies to the other steering methods, but it seems fairly tedious and computationally expensive to perform layer selection by sweeping over all activation layers and assessing the steering effects. 2. A minor point: in the introduction L61, "our method allows the model to 'speak up'". This analogy was slightly confusing to me which made it difficult to grasp the idea of the method when reading it. The design of the objective became clear after reading the method section, however, I still don't understand how the method allows the model to "speak up".

Questions

1. How much improvement does the bi-directional coefficient bring? Does the uni-directional version (equation 2) work well on its own? 2. Does steering different layers produce a behavioral difference in the LLM responses? And are there any insights into why the 15th layer works best? 3. Has the author tried steering multiple layers and does it bring improved performance? 4. Minor: - maybe change iteration symbol $T$ with something else? It also appears in $r^i_T$ as the target. - add a brief description of the parameter $\beta$ to the algorithm inputs.

Rating

8

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors have included a discussion in Appendix H.

Summary

This paper introduces a novel method called Bi-directional Preference Optimization (BiPO) for producing more effective steering vectors to control the behavior of large language models (LLMs). It includes the demonstration of personalized control over model behaviors by adjusting vector direction and magnitude.

Strengths

1. Extensive experiments show effectiveness across various tasks including steering AI personas, managing truthfulness, mitigating hallucinations, and addressing jailbreaking attacks. 2. The BiPO approach is novel in how it optimizes steering vectors using contrastive preference data to directly influence generation probabilities, rather than relying solely on activation differences. 3. The paper is generally well-written and structured logically.

Weaknesses

1. While the empirical results are strong, the paper lacks rigorous theoretical analysis of why the BiPO approach works better than baselines. A more formal treatment of the optimization objective could strengthen the work. 2. The paper does not thoroughly discuss the computational requirements of the method compared to baselines. More details on training time and resources needed would be helpful for assessing practical applicability. 3. Experiments are insufficient due to the lack of different and novel jailbreak methods.

Questions

1. How does the computational cost of BiPO compare to baseline methods? What are the trade-offs in terms of performance vs. efficiency? 2. The paper focuses primarily on successful cases. A more in-depth analysis of scenarios where the method underperforms or fails could provide valuable insights.

Rating

5

Confidence

4

Soundness

2

Presentation

3

Contribution

2

Limitations

While the authors discuss some ethical considerations, more thorough analysis of how the technique could potentially be misused (e.g., for generating misinformation) would strengthen the paper.

Summary

This paper focuses on how to better steer the behavior of LLMs. Specifically, they followed the activation engineering method and introduced the so-called "bi-directional preference optimization" to create more effective steering vectors. Unlike previous activation addition methods that rely on the differences in activations from contrastive prompts to compute the steer vectors, this approach directly optimizes steering vectors to influence the generation prob of contrastive data pairs w/ different human preferences. In this way the model outputs different generations conditioned on the steering vector and its scale. This method allows for personalized control over LLM outputs in alignment-related scenarios such as managing truthfulness and defending against jailbreaking attacks. The paper also highlights the transferability of these vectors across different models and showcases the synergistic benefits of applying multiple vectors simultaneously.

Strengths

1. The introduction of bi-directional preference optimization, which directly optimizes the steering vectors rather than computing them in a statistical ad-hoc way to create steering vectors is novel. It offers a more precise representation of target behaviors compared to previous methods. 2. The methodology is well-explained, with clear steps and detailed descriptions of the experimental settings and results. 3. The ability to personalize or steer LLMs effectively is indeed important, practically and theoretically.

Weaknesses

1. The activation engineering has many heuristic designs in it and lacks solid formulation. The work states that it's inspired by DPO for direct optimization, yet has way much empirical stuff. 2. The 'bi-directional preference optimization' may be complex to implement and more costly compared to other activation engineering methods, potentially limiting its accessibility. 3. While the paper shows positive results on specific models and tasks, it is unclear how well the method generalizes to other LLMs or less common tasks. 4. The approach's scalability to larger models or more extensive datasets was not thoroughly explored, which could be a limitation for practical applications.

Questions

Personalization is indeed important, however, the activation engineering methods seem to have many heuristic things involved such as the position of the token of steering vectors the statistical objective to compute it, and so on, on which this work is developed. How about the methods' performance compared with other personalized steering LLM methods?

Rating

6

Confidence

4

Soundness

2

Presentation

2

Contribution

3

Limitations

It would be beneficial to discuss the method's scalability in more detail and explore potential challenges in applying this approach to even larger models or datasets.

Reply by Reviewer

Thank you for your detailed rebuttal. Your response has addressed my major concerns. Based on this additional information, I plan to increase my score for your submission. I will also engage in further discussion with the other reviewers to ensure we consider your clarifications.

Nice rebuttal!

Thank you for the detailed replies to my questions. I have raised my score from 5 to 6.