Improving Energy Efficiency in Femtocell Networks: A Hierarchical Reinforcement Learning Framework

This paper investigates energy efficiency for the two-tier femtocell networks through combining game theory and stochastic learning. With the Stackelberg game formulation, a hierarchical reinforcement learning framework is developed to study the joint expected utility maximization of macrocells and femtocells. The macrocells behave as the leaders and the femtocells are followers during the learning procedure. At each time step, the leaders commit to dynamic strategies based on the best responses of the followers, while the followers compete against each other with no further information but the leaders' strategy information. In this paper, two learning algorithms are proposed to schedule each cell's transmission power. Numerical results are presented to validate the proposed studies and show that the two learning algorithms substantially improve the energy efficiency of the femtocell networks.