A Reinforcement Learning Approach for D2D-Assisted Cache-Enabled HetNets

Abstract

Mobile edge caching (MEC) and device to device (D2D) communications are two potential technologies to resolve traffic overload in heterogeneous networks. Prior works usually investigate them separately with MEC for traffic offloading and D2D for information transmission. In this paper, a composite framework consists of MEC and cache-enabled D2D communications is proposed to minimize the energy cost of systematic traffic transmission, where file popularity and user preference are the critical criteria for small base stations (SBSs) and users respectively. Under this framework, we propose a novel caching strategy where Markov decision process (MDP) is applied to model the requesting behaviors of users. A new algorithm based on reinforcement learning (RL) is proposed to reveal the popularity of files as well as users' preference. In particular, Q-learning (QL) algorithm and deep Q- network (DQN) algorithm are respectively applied to users and SBS due to different complexity of status. To save the energy cost of systematic traffic transmission, users acquire partial traffic through D2D communications based on the cached contents. Taking the memory limits, D2D available files and status changing into consideration, the proposed RL algorithm enables users' devices and SBS to prefetch the optimal files while learning, and hence reducing the energy cost significantly. Simulation results demonstrate the superior energy saving performance of the proposed RL-based algorithm over other existing methods under various conditions.