Summary
The paper proposes MEET, a method to train a LLM to generate "good" and "bad" answers to a given question / task by conditioning the model computation with an adapter (LoRA or Soft Prompt). To do so, they adopt a two-step training procedure. First, they train a "good control adapter" and a "bad control adapter" on good answers and bad answers respectively while keeping the base LM fixed, then they fine-tune both the control adapters and the base model. The authors show that this two step procedure is important to achieve gains over the Chain of Hindsight baseline (basically a baseline where control adapters is just a handcrafted prompt "A good/bad conversation is:") and DPO on two datasets OpenAI Summary and HH-RLHF from Anthropic.
Strengths
- The paper is well-written, the details of the experimental setting are clear.
- The two-stage training procedure is interesting and its importance is validated by the ablation study.
- Results seem to suggest that the two-step optimization method delivers gains w.r.t. DPO.
Weaknesses
- It feels like the authors are a bit confused on where the novelty of their paper really lies, they seem to suggest that it is in using adapters to control generation, but imho, the interesting bit is more on the two-step training procedure that guarantees information is captured by the adapters and thus they are not "information-starved" by the full LM fine-tuning (easy to fix)
- The more problematic bit is that authors' confusion seems to have affected the overall experimental methodology; for example, the authors seem to tie their method to the specific loss function used (i.e. MLE) and compare to DPO, while their method can be used on top of DPO. Moreover, the baselines numbers are a bit concerning and some important baselines are missing (overall harder to fix)
Questions
About novelty:
The paper proposes to learn "attributes" conditional models with adapters, which have been proposed in https://arxiv.org/pdf/2302.08453.pdf for diffusion models for example. So, here, the novelty might reside in 1/ applying this general idea to textual generation tasks and 2/ the two-stage training approach proposed to train these adapters. The current stance of the paper is that the main novelty is to apply LoRA adapters for generation instead of hard prompts. I feel like 2/ is a more interesting and impactful contribution but currently it is a bit understated in the paper, so it feels like it should be the central focus of the paper. I feel like the paper can be an interesting set of experiments showing that the two-stage approach prevent adapters from being "information-starved" from full model fine-tuning.
About experiments:
Confusion about the contributions seem to appear in Section 3, where the authors tie their method to MLE loss (1) and (2) and compare in the experiments with a DPO baseline. This is a bit surprising to me given that their method can be deployed on top of DPO, i.e. Eq (1) and (2) can use DPO instead of MLE (to train each good and bad expert), so I am not sure why DPO would be a baseline in the experiments. On the contrary, I would have expected to see two versions of their method in the experiments: with Eq. (1) and (2) using DPO (MEET-DPO) and Eq. (1) and (2) using MLE (MEET-MLE).
From all experiments, one straightforward baseline is missing in addition to CoH: SFT -- which just trains on positive data.
Similarly, for MEET-MLE, what is the impact of integrating negative data? i.e. what is the gap between MEET-SFT, which just trains the controllable adapter of positive data and MEET-MLE, which trains on both positive and negative data with Eq. 2?
In the first dataset, DPO underperforms CoH on OpenAI/Summary dataset. The fact that DPO underperforms CoH on this dataset is a bit suspicious. Did you tune the \beta parameter for DPO on both datasets ?
How do you do cross-validation in these two datasets? Are you searching for the best HPs for each method on the validation set?
Taken together, your results currently show that DPO is useless in these two datasets and severely underperform MLE training with MEET. I am not sure this result can be published without further ablations and baselines as I suggest above, especially it appears to me that MEET can be further improved with DPO training.
Please, do not consider my score as final, I am willing to increase the score substantially if the authors can give answers to my questions.
Rating
3: reject, not good enough
Confidence
4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.