DHRL-FNMR: An Intelligent Multicast Routing Approach Based on Deep Hierarchical Reinforcement Learning in SDN

The multicast routing problem in software-defined networking (SDN) is an NP-hard problem. The existing solution methods based on deep strength learning suffer from the problems of branch redundancy, an excessively large action space and slow convergence of the intelligent models. In this paper, an intelligent multicast routing algorithm based on deep hierarchical reinforcement learning is proposed to circumvent the aforementioned problems. First, the optimal multicast tree problem is decomposed into two subproblems: fork node selection and the construction of an optimal path from a fork node to a destination node. Second, a multichannel matrix is designed as the state space for the internal and external controllers of hierarchical reinforcement learning based on the global network-aware information characteristics of SDN. Then, different action spaces are designed for the upper and lower subproblems, four action selection policies are designed for constructing multicast paths, and different reward policies are designed at different levels. Finally, a series of experiments and their results show that the designed algorithm not only searches the multicast tree efficiently but also converges faster and without redundant branches, with better performance in terms of bandwidth, delay and packet loss rate than the current mainstream solution algorithms. The codes for DHRL-FNMR are open and available at https://github.com/GuetYe/DHRL-FNMR.