Summary
This work offers an alternative derivation of Direct Preference Optimization using Markov Decision Processes (MDPs) that enables to define token-level rewards. This has theoretical and practical applications. Fundamentally that DPO can learn token level rewards and assign different credit to tokens, a relation between guided decoding and search-based algorithms and an explanation of the phenomenon of decreasing likelihoods during DPO training.
The core of the paper is a derivation that follows closely that of DPO but from an MDP perspective i.e. [S4.1]
1. Determining the optimal policy for the KL-controled reward maximization loss. Compared to DPO here they consider a more general case of the loss that uses a MDP characterization with states and actions as a function of time. The optimal policy for this case is a know result, as it happened with DPO.
2. Deriving a the reward from the this optimal policy. Compared to DPO, here the use of the Bellman Equations is needed to obtain the token level relations. The value function cancels in the same way the partition function cancels in DPO. I may have missed details on this part, but it seems solid. Result of this is Eq (9) a derivation of DPO as an MDP.
3. In addition to this they generalize the definition of equivalency of DPO for the MDP case. This exploits very similar properties as in the case of DPO (S4.2).
In the following sections they use these derivations to deduce certain properties of DPO and empirically confirm these i.e.
4. The MDP derivation of DPO, indicates that it can learn token level rewards.
5. Guided decoding that maximizes a reward, also based in the MDP formulation, is the same as maximum likelihood DPO optimization.
6. Explain the known phenomena that "the implicit rewards of both the chosen and rejected response decline, though the margin between them increases", which once is seen from the perspective of the MDP it is explainable.
Reasons to accept
This paper provided novel and relevant insights into DPO, in particular
1. a theoretical derivation token-level reward DPO bringing concepts from Markov Decision Processes.
2. showing that DPO maximum likelihood search is equivalent to decoding to maximize a reward function.
3. provide insight and empirical results showing the importance of the starting policy initialization in DPOs training.
Reasons to reject
Not really reasons to reject, but more clarity on some RL terms NLP people may not be fully familiar with. Thanks to RLHF's impact in LLMs, the average reader on this conference will be familiar with a lot of the material (I am one of those), but maybe some concepts may remain unfamiliar, such as, bandits, MDP and Q function.
I have to say I did not fully understand some of the claims and some aspects seem self-evident, I put these on the question section. Probably after discussion this should all be ok.
Questions to authors
### S 4.1
>While the original derivation of DPO relies on the fact that Q∗(x, y) = r(x, y)
*Question 1:* Can you clarify this assertion? The original derivation in (Rafailov et al 2023) does not use MDPs or Q. It basically consists on two elements
- derivation of the form of the optimal policy for the KL-controlled reward maximization objective (appendix A.1). This just needs algebraic manipulation of the original objective and checking the solution satisfies this and is unique.
- derivation of the DPO objective, which follows again by algebraic manipulation of the optimal distribution derived above and replacing this in the Bradly-Terry model.
I am aware that there are earlier derivations for the optimal policy, but this one seems simple and not need of the assumption above.
### S4.1
>This style of relationship was first explored by Garg et al. (2022) in imitation learning and later in Hejna & Sadigh (2024) for preference-based RL. However, these works require the use of a discount factor γ < 1 which is typically not used in RLHF.
*Question 2:* Could you expand on this difference? It seems to be and important aspect to judge the novelty. Maybe add some summary of the derivation used in those works in the Appendix A.
### S.1
>Second, we demonstrate that likelihood search over a DPO model is analogous to searchign over a reward function during decoding as done by contemporary works ().
*Question 3:* can you provide this missing reference?
### S 4.2
>Interestingly, in practice, the potential function Φ(st) represents the free parameter in the logits of the language model. An equal shift along all logits yields the same policy, but different Q-functions and corresponding rewards. The above Theorem proves that all of these are in the same equivalence class and induce the same set of preferences.
I fail to see why this is not self-evident. Scaling the logits by f(x) does not affect the probability distribution. Could you please clarify this?