In this paper, we develop a recommender system for a game that suggests\npotential items to players based on their interactive behaviors to maximize\nrevenue for the game provider. Our approach is built on a reinforcement\nlearning-based technique and is trained on an offline data set that is publicly\navailable on an IEEE Big Data Cup challenge. The limitation of the offline data\nset and the curse of high dimensionality pose significant obstacles to solving\nthis problem. Our proposed method focuses on improving the total rewards and\nperformance by tackling these main difficulties. More specifically, we utilized\nsparse PCA to extract important features of user behaviors. Our\nQ-learning-based system is then trained from the processed offline data set. To\nexploit all possible information from the provided data set, we cluster user\nfeatures to different groups and build an independent Q-table for each group.\nFurthermore, to tackle the challenge of unknown formula for evaluation metrics,\nwe design a metric to self-evaluate our system's performance based on the\npotential value the game provider might achieve and a small collection of\nactual evaluation metrics that we obtain from the live scoring environment. Our\nexperiments show that our proposed metric is consistent with the results\npublished by the challenge organizers. We have implemented the proposed\ntraining pipeline, and the results show that our method outperforms current\nstate-of-the-art methods in terms of both total rewards and training speed. By\naddressing the main challenges and leveraging the state-of-the-art techniques,\nwe have achieved the best public leaderboard result in the challenge.\nFurthermore, our proposed method achieved an estimated score of approximately\n20% better and can be trained faster by 30 times than the best of the current\nstate-of-the-art methods.\n