Distributed Multi-Hop Traffic Engineering via Stochastic Policy Gradient Reinforcement Learning

Abstract

Multi-hop networks (e.g., mesh, ad-hoc, and sensor networks) are important and cost-efficient communication backbones. Over the last few years wireless data traffic has drastically increased due to the changes in the way today's society creates, shares, and consumes information. This demands the efficient and intelligent utilization of limited network resources to optimize network performance. Traffic engineering (TE) optimizes network performance and enables optimal forwarding and routing rules to meet the quality of service (QoS) requirements for a large volume of traffic flows. This paper proposes a distributed model-free TE solution based on stochastic policy gradient reinforcement learning (RL), which aims to learn a stochastic routing policy for each router so that each router can send a packet to the next-hop router according to the learned optimal probability. The proposed policy-gradient solution naturally leads to multi-path TE strategies, which can effectively distribute the high traffic loads among all available routing paths to minimize the E2E delay. Moreover, a distributed software-defined networking architecture is proposed, which enables the fast prototyping of the proposed multi-agent actor-critic TE (MA-AC TE) algorithm and in-nature supports automated TE through multi-agent RL learning.