Discovering Preference Optimization Algorithms with and for Large Language Models

Summary

This paper introduces a method of searching for offline RL objectives by using LLMs to generate and refine objectives. They demonstrate that several objectives discovered using this method are able to achieve higher evaluation scores than existing objectives (e.g. DPO) on a variety of benchmark tests. They provide a high level analysis of one discovered objective, hypothesizing that it may possess desirable properties that lead to these performance boosts.

Strengths

The paper is well written, organized, and easy to understand. While the use of LLMs to modify prompts and self-improve in other ways has been studied, using them to improve objective functions is a novel idea that the authors demonstrate holds potential. The authors demonstrate that the objectives discovered can be useful, showing improvements on a variety of generation and classification tasks.

Weaknesses

The main weakness of this paper in my opinion is that it isn't clear what results the paper is trying to present. While this is partially an issue of organization, it is also an issue with the amount of evidence given to the claims in the paper. If these was made clearer, I would be inclined to raise my score. I detail below: 1. The paper starts by introducing a new method for discovering new optimization objectives using LLMs, then transitions quite abruptly to describing the properties of DiscoPop. While both could be valuable contributions, I don't feel that either is given quite the attention required. It would help to clarify what the main focus of the paper is. If DiscoPOP is the focus, then claims about its properties should be better studied. If the discovery method is the focus, more emphasis should be placed on the method's performance and ability to produce novel and useful objectives. 2. The majority of insights about DiscoPOP are presented as hypotheses or intuitive insights (e.g. the non-convex portion, and how the model behaves at $\pm\inf$. This is not enough to support the claim made in the conclusion that they provide insights into what properties an optimal objective function should possess. While it \emph{may} be enough to say the provide intuition into why DiscoPOP is successful, further experimentation would be necessary to make optimality claims like this. 3. Some portions of the discovery method are vague as well. Though the authors do not observe the models simply regurgitating old objectives and cite a pattern of objective refinement for supervised learning, this is of course not guaranteed, and not well studied. Further experiments or details on the how often they observe refinement vs regurgitation would help to support the utility of the method.

Questions

1. How often are unit tests for the generated code failed?

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

The limitations described cover most points, including the difficulties and uncertainties associated with using LLMs to improve upon objectives in a rigorous fashion.

Summary

The paper proposes DiscoPOP, an algorithm for discovering preference optimization loss functions using Large Language Models (LLMs). The authors propose an LLM-driven objective discovery process by iterative prompting LLMs by previously evaluated performance metrics. Experiments on various benchmarks demonstrate its effectiveness.

Strengths

1. This method is innovative. The idea of updating the loss function through automated exploration is both interesting and novel. The process being fully automated and end-to-end makes it particularly neat. 2. The results are impressive. DiscoPOP demonstrates strong performance across various benchmarks, showing its potential.

Weaknesses

I think the updating process might be sensitive to the prompts used for proposing new loss functions.

Questions

Refer to the weakness

Rating

6

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

Refer to the weakness

Summary

The paper proposes an algorithm to discover preference objective functions using LLM for LLM preference optimization. Authors conduct experiments with the discovered objectives on multiple datasets and demonstrate that the discovered objective functions generally perform better than baselines. The authors also show interesting insight into the discovered objective function, LRML.

Strengths

1. The paper introduces an interesting and new LLM-driven objective discovery algorithm to search for good objective functions using LLMs for the preference optimization of LLMs. 2. The method successfully finds an objective function that can generally offer a better performance on multiple benchmarks, indicating the effectiveness of the discovery method. 3. The authors show interesting insights into the best objective function found by the proposed method.

Weaknesses

1. The objective function is discovered with a different placement of $\beta$ than the objective function used for evaluation, which causes a misalignment of the discovery and the evaluation process. It would be nice to keep them aligned. 2. The LLM in the discovery process is a clever generator for searching candidates. It is unclear if the LLM's capability used in the discovery process can change the duration for finding a good objective function or if it will cause the search process to fail. 3. Similar to the point above, it would be nice to test some non-LLM-based generators of the objective function and perform some traditional searching algorithms such as evolutionary search to demonstrate the effectiveness of the LLM.

Questions

See weakness above.

Rating

7

Confidence

3

Soundness

3

Presentation

3

Contribution

3

Limitations

See weakness above.

Summary

This paper proposes a novel approach to improving LLMs by using an automated system to discover new optimization algorithms. Traditionally, enhancing LLMs has relied heavily on hand-designed loss functions, but this research employs an LLM to iteratively generate and refine these functions itself. This paper introduced DiscoPOP, which blends logistic and exponential losses to outshine existing methods. This algorithm was evaluated across a range of tasks, including multiturn dialogue, sentiment generation, and summarization, where it consistently delivered good results.

Strengths

- The automatic exploration and evaluation pipeline is promising and it can discover new algorithms without human intervention. - By using the above pipeline, the new discovered algorithm in preference optimization achieved SOTA, proving its effectiveness. - This can contribute to lots of other area and let machines themselves to discover and evaluate new efficent algorithms.

Weaknesses

Although this paper mianly focuses on preference optimization task, the dicovery method seems to be easily to adapt to other domains/tasks. A general concern with the paper is its reliance on a complex method that involves detailed tuning of hyperparameters. This complexity could limit the method's broader applicability unless the process can be generalized or adapted effectively to other contexts. However, the paper does not thoroughly address how to establish and optimize the discovery pipeline across different scenarios, which could hinder its practical utility. More guidance on adapting the methodology to a variety of use cases would significantly enhance its value and impact.

Questions

1. How well does the proposed LLM-driven discovery method generalize to other domains/tasks beyond preference optimization? Are there specific modifications needed to adapt this method to other fields? 2. The authors mention that DiscoPOP blends logistic and exponential losses. Could you elaborate on the theoretical justification for this choice? How does this combination affect the convergence properties of the algorithm? 3. Considering the non-convex nature of DiscoPOP, what strategies do you suggest to avoid local minima during optimization?

Rating

7

Confidence

4

Soundness

3

Presentation

3

Contribution

3

Limitations

The authors discussed the limitations of DiscoPOP.