LLM-Empowered Creator Simulation for Long-Term Evaluation of Recommender Systems Under Information Asymmetry

Abstract

Maintaining the long-term sustainability of recommender systems (RS) is crucial. Traditional RS evaluation methods primarily focus on the user's immediate feedback (e.g., click), however, they often overlook the long-term effect involved by the content creators. In the real world, content creators can strategically create and upload new items to the platform by analyzing users' feedback and preference trends. Although previous studies have attempted to model creator behaviors, they often overlook that such behaviors are under conditions of information asymmetry. This asymmetry arises because creators mainly access the user feedback on the items they produce, while the platform has access to the full spectrum of feedback data. However, existing RS simulators often fail to consider such a condition, making the long-term RS evaluation inaccurate. To bridge this gap, we propose a Large Language Model (LLM)-empowered creator simulation agent named CreAgent. By utilizing the belief mechanism from game theory and the fast-and-slow thinking framework, we can simulate the creator's behaviors well under information asymmetry. Furthermore, to enhance CreAgent's simulation ability, we utilize Proximal Policy Optimization to fine-tune CreAgent. Our credibility validation experiments demonstrate that our simulation environment effectively aligns with the behaviors of real-world platforms and creators, thereby enhancing the reliability of long-term evaluations in RS. Furthermore, leveraging this simulator, we can examine whether RS algorithms, such as fairness- and diversity-aware methods, contribute to improving long-term performance for different stakeholders.