Recommender systems aim to leverage user interaction signals to recommend items that users are likely to be interested in. Motivated by the success of Large Language Model (LLMs), generative recommendation (GR) has recently gained increasing attention, typically following a two-stage paradigm: supervised fine-tuning followed by preference alignment. However, aligning generative recommenders with users' personalized preferences remains challenging, as user feedback is inherently heterogeneous and uncertain. Different types of user interaction signals reflect varying levels of intent and should therefore be modeled differently. In this work, we propose Adaptive Mix Preference Optimization (AMPO), an adaptive alignment framework that mixes likelihood and preference objectives with self-calibrated, sample-wise confidence adjustment. AMPO introduces an adaptive target margin that leverages the model's own probability ratio to modulate optimization strength: confident pairs receive full margins that reinforce correct rankings, while uncertain pairs receive reduced margins that prevent overfitting to ambiguous signals. Additionally, AMPO incorporates negative log-likelihood regularization on preferred items to counteract likelihood displacement, a phenomenon where contrastive objectives cause preferred and non-preferred probabilities to collapse simultaneously. Such a design eliminates the need for a reference model, yielding up to 2.5x speedup and 35% memory reduction. Extensive experiments on public benchmarks and a large-scale industrial dataset demonstrate consistent improvements in ranking metrics. Online A/B tests on a major e-commerce platform further confirm statistically significant gains in click-through and conversion rates. The code is available at https://github.com/jumbo-q/ampo.
Paper
Full text
Adaptive Mix Preference Optimization for Generative Recommendation
Semantic Scholar · 2026
Abstract
Recommender systems aim to leverage user interaction signals to recommend items that users are likely to be interested in. Motivated by the success of Large Language Model (LLMs), generative recommendation (GR) has recently gained increasing attention, typically following a two-stage paradigm: supervised fine-tuning followed by preference alignment. However, aligning generative recommenders with users' personalized preferences remains challenging, as user feedback is inherently heterogeneous and uncertain. Different types of user interaction signals reflect varying levels of intent and should therefore be modeled differently. In this work, we propose Adaptive Mix Preference Optimization (AMPO), an adaptive alignment framework that mixes likelihood and preference objectives with self-calibrated, sample-wise confidence adjustment. AMPO introduces an adaptive target margin that leverages the model's own probability ratio to modulate optimization strength: confident pairs receive full margins that reinforce correct rankings, while uncertain pairs receive reduced margins that prevent overfitting to ambiguous signals. Additionally, AMPO incorporates negative log-likelihood regularization on preferred items to counteract likelihood displacement, a phenomenon where contrastive objectives cause preferred and non-preferred probabilities to collapse simultaneously. Such a design eliminates the need for a reference model, yielding up to 2.5x speedup and 35% memory reduction. Extensive experiments on public benchmarks and a large-scale industrial dataset demonstrate consistent improvements in ranking metrics. Online A/B tests on a major e-commerce platform further confirm statistically significant gains in click-through and conversion rates. The code is available at https://github.com/jumbo-q/ampo.