Reverse Training to Nurse the Reversal Curse

Summary

The paper present the problem of reversal curse and a simple approach to help alleviate the issue. The reversal curse is the phenomenon that LLMs seem to lack the capability to properly understand relations such as "Paris is the capital of France", demonstrated by the models' inability to infer from the previous that "The capital of France is Paris". The proposal made in the paper is to pre-train or fine-tune LLMs with data containing reversed text (not just standard left-to-right). The paper investigates multiple approaches to revering text and evaluates model performance in both reversal comprehension and standard LLM evaluation benchmarks. The finding is that models trained with reversed text perform better in the reversal tasks and the performance on standard evaluation benchmarks does not deteriorate, indicating that the approach would be a suitable method for alleviating the reversal curse. The paper is well written, well structured and contains good selection of evaluation results. The only bigger issue with the paper is the lack of qualitative evaluation of the model outputs. It is well known that many of the evaluation benchmarks and tasks can be solved by models that do not really "understand" the language or the task or by models that otherwise generate poor quality language. Including qualitative evaluation and example model outputs would help the reader understand what effect the reverse training has on the model.

Rating

6

Confidence

4

Ethics flag

1

Reasons to accept

The paper presents a simple but effective data augmentation method that can help alleviate the reversal curse in LLMs. The paper is well written, the main arguments are well presented and the experimental results support the arguments.

Reasons to reject

Intuitively, training on reversed text should harm the model in terms of how natural the output of the model is. In addition to simple benchmark test results, I would have liked to see some more qualitative evaluation showing what the impact of reverse training has on the model output.

Questions to authors

I would like to see more qualitative evaluation of model output, comparing baseline and revere trained versions.

Summary

Motivation of the paper is that language models do not learn the reverse of facts (e.g., "X has feature Y" versus "Y is a feature of X") due to their left-to-right training and the pre-training data including facts in only one of those two ways. This causes models to not be able to answer some questions based on how it's asked (e.g., "What is a feature of X?" versus "What is Y a feature of"), even though the fact is in its training dataset. - The symbolic reverse task allows a controlled study of the approach and provides clear validation of the authors' idea. - The "celebrity task" has very low accuracy numbers, seemingly too low to make any conclusions. The data appears to be simply insufficient for the model to perform this task. - Real-world larger-scale tasks show that using the reverse training approach during pre-training and/or fine-tuning can be helpful. When the same amount of training data is used (data-matched) the proposed approach is able to solve previously unsolvable tasks, with pretty good accuracy numbers. The best approach doesn't seem to require the reverse training during pre-training but just during fine-tuning.

Rating

7

Confidence

4

Ethics flag

1

Reasons to accept

The authors present a very neat idea that has merit and can be very impactful as it is easy to apply while training language models. Their presentation is very clear with extra details provided in the Appendix. Results are pretty impressive given the simplicity of the approach.

Reasons to reject

A direct empirical comparison with Guo et al 2024 is needed to determine the value of this approach. While it is a more complex augmentation and therefore can only be applied at fine-tuning stage, I am not convinced by the mixed results that doing the augmentation during pre-training will provide the best results in a real-world scenario. Tables 6 and 7 imply that the best approach would be to do standard pre-training and reverse fine-tuning. In that case, I would've liked to see a comparison with Guo et al's work where both approaches are applied during fine-tuning only.

Questions to authors

- Do you think this would work for other languages? Given multilingual nature of LLMs, some discussion of this would be good. - typo in Section 4 "Mitgiations" - I wonder if the "reversal curse" is just a symptom of a deeper issue with language models' logical reasoning abilities. While LLMs are known to reason over the facts they've been trained on, this could be just one of many reasoning tasks they are bad at. I would encourage the authors to provide their thoughts on this.

Summary

This paper proposes to use reverse training to mitigate the reversal curse in language models. In short, language models have been shown by prior work to present a "reversal curse": models are typically presented with strings of the form, e.g. "Paris is the capital of France", and they struggle to answer questions such as "what is the capital of France?"; this is because they are rarely presented with strings where "Paris" follows "France" (as in, e.g., "in France's capital, Paris, ..."). The authors then show that, when language models are trained on a mix of forward and reverse strings (using a standard language modeling next token prediction objective), LMs perform much better on questions which require such a "reverse answer".

Rating

6

Confidence

3

Ethics flag

1

Reasons to accept

The paper investigates an interesting topic: why do language models perform better on facts written with a certain directionality than another. This paper also performs a large number of analyses related to the impact of reverse training on model performance. It studies the impact of reverse training in: a toy task, pretraining, and fine-tuning. Further, it considers the impact of reversing strings at the token, word, or entity level. Finally, it studies the impact of reverse training not only on tasks in which the model originally performs poorly, but also investigates whether reverse training hurts model performance on tasks the model originally did well in.

Reasons to reject

While reverse training seems to experimentally help with the reversal curse issue, it seems to focus on solving a "symptom" caused by how language models are trained/defined rather than trying to fix the source of the issue. Identifying the source of why language models are inconsistent in representing facts based on the directionality in which they are presented, and proposing a principled way to fix it, would make this paper's contribution stronger in my opinion.

Generalizing to other languages

Thank you for the clarifications. I do want to dig into the multilingual aspect a bit more, as I think that the merit of this approach is mainly for practitioners (as you also stated in your rebuttal), and that depends on how well the method generalizes beyond English. In your rebuttal to my fellow reviewer, you said "In a structural sense, the original and reversal data have significantly different syntax, even if they have the same words, so that the model can learn to distinguish them easily." What if the language has free word order? Would that not violate your assumption that the original and reversal data have significantly different syntax, therefore the model would be able to implicitly learn a "switch"?

Response to Authors

I thank the authors for their response. I've read both the other reviews, and the authors' responses, and I am keeping my scores.