A Reinforcement-Learning-Enhanced LLM Framework for Automated A/B Testing in Personalized Marketing

For personalized marketing, a new challenge of how to effectively algorithm the A/B testing to maximum user response is urgently to be overcome. In this paper, we present a new approach, the RL-LLM-AB test framework, for using reinforcement learning strategy optimization combined with LLM to automate and personalize A/B tests. Design details RL-LLM-AB test is built upon the pre-trained instruction tuning language model, and it firstly generates A/B versions of candidate content variants with the Prompt-Conditioned Generator, and finally, it dynamically embeds and fuses the user portrait and the context of the current query with the multi-modal perception module to constitute the current interaction state. Then the content version is assigned in real-time through the policy optimization module with Actor-Critic structure, and the long-term revenue is estimated according to real-time feedback (such as click-through rate and conversion rate). Furthermore, a Memory-Augmented Reward Estimator is embedded into the framework to capture the long-term user preference drift which help to generalize policy across the multiple users and content context. Numerical results demonstrate the superiority of our proposed RL-LLM-ABTest over existing B tests based on the classical A/B testing, Contextual Bandit, and benchmark reinforcement learning methods on real-world marketing data.